[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:22:49 dist-test-slave-gw2v krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:22:49 dist-test-slave-gw2v krb5kdc[399](info): set up 2 sockets
Sep 27 03:22:49 dist-test-slave-gw2v krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 03:22:50.195011 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:22:50 dist-test-slave-gw2v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:22:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:22:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:22:50.309309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:50.387756 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:50.387977 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:50.388006 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:50.389212 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:22:50.389263 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:50.389281 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:50.389297 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:22:50.389310 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:22:50.390811 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:50.391117 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:50.391363 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:22:50.397096 412 server_base.cc:1034] running on GCE node
W20260927 03:22:50.397048 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:50.397075 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:50.397027 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:50.397598 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:50.397936 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:50.399096 412 hybrid_clock.cc:648] HybridClock initialized: now 1790479370399070 us; error 36 us; skew 500 ppm
Sep 27 03:22:50 dist-test-slave-gw2v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:50.401593 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:22:50.402046 412 webserver.cc:460] Webserver started at http://127.0.98.62:38489/ using document root <none> and password file <none>
I20260927 03:22:50.402292 412 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:50.402350 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:50.402482 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:50.403644 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "abf6bd27e740419c92dd877fec7429ce"
format_stamp: "Formatted at 2026-09-27 03:22:50 on dist-test-slave-gw2v"
server_key: "da389f66f50ad60000dc53a058cf3663"
server_key_iv: "dda442ac763e70521387683f77de2678"
server_key_version: "encryptionkey@0"
I20260927 03:22:50.403992 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "abf6bd27e740419c92dd877fec7429ce"
format_stamp: "Formatted at 2026-09-27 03:22:50 on dist-test-slave-gw2v"
server_key: "da389f66f50ad60000dc53a058cf3663"
server_key_iv: "dda442ac763e70521387683f77de2678"
server_key_version: "encryptionkey@0"
I20260927 03:22:50.405303 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:22:50.405977 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:50.406123 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:22:50.406215 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "abf6bd27e740419c92dd877fec7429ce"
format_stamp: "Formatted at 2026-09-27 03:22:50 on dist-test-slave-gw2v"
server_key: "da389f66f50ad60000dc53a058cf3663"
server_key_iv: "dda442ac763e70521387683f77de2678"
server_key_version: "encryptionkey@0"
I20260927 03:22:50.406275 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:50.478680 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:50.480860 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:50.481005 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:50.484699 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37741
I20260927 03:22:50.484740 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37741 every 8 connection(s)
I20260927 03:22:50.485061 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:22:50.485893 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:50.488199 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce: Bootstrap starting.
I20260927 03:22:50.488725 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:50.488966 480 log.cc:824] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:50.489598 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce: No bootstrap required, opened a new log
I20260927 03:22:50.491186 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf6bd27e740419c92dd877fec7429ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37741 } }
I20260927 03:22:50.491325 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:50.491348 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf6bd27e740419c92dd877fec7429ce, State: Initialized, Role: FOLLOWER
I20260927 03:22:50.491484 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf6bd27e740419c92dd877fec7429ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37741 } }
I20260927 03:22:50.491541 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:22:50.491564 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:22:50.491607 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:50.492141 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf6bd27e740419c92dd877fec7429ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37741 } }
I20260927 03:22:50.492226 480 leader_election.cc:302] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abf6bd27e740419c92dd877fec7429ce; no voters:
I20260927 03:22:50.492365 480 leader_election.cc:288] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:22:50.492416 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:22:50.492576 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:22:50.492597 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [term 1 LEADER]: Becoming Leader. State: Replica: abf6bd27e740419c92dd877fec7429ce, State: Running, Role: LEADER
I20260927 03:22:50.492679 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf6bd27e740419c92dd877fec7429ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37741 } }
I20260927 03:22:50.493168 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader abf6bd27e740419c92dd877fec7429ce. Latest consensus state: current_term: 1 leader_uuid: "abf6bd27e740419c92dd877fec7429ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf6bd27e740419c92dd877fec7429ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37741 } } }
I20260927 03:22:50.493250 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:50.493283 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abf6bd27e740419c92dd877fec7429ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf6bd27e740419c92dd877fec7429ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37741 } } }
I20260927 03:22:50.493360 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:50.493475 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:22:50.494058 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:22:50.494908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 412
I20260927 03:22:50.495004 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:22:50.495489 491 catalog_manager.cc:1347] Generated new cluster ID: 420d44ee4ba645419f96fd14672ee49e
I20260927 03:22:50.495530 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:22:50.495754 392 external_mini_cluster.cc:1428] Setting key f012b54cdf20fc2a2af6798a72e51c49
I20260927 03:22:50.505381 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:22:50.506021 491 catalog_manager.cc:1504] Loading token signing keys...
Sep 27 03:22:50 dist-test-slave-gw2v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:22:50.513007 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:50.496669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.0.98.62:37741)
0927 03:22:50.496826 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:50.496834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:50.497097 (+ 263us) server_negotiation.cc:407] Connection header received
0927 03:22:50.499154 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:50.499162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:50.499218 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:50.499361 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:50.501247 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:50.502762 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:50.504212 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:50.504832 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:50.504939 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:50.509232 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:50.509237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:50.509237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:50.509248 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:50.511123 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:50.511619 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:50.511622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:50.511622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:50.511725 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:50.511981 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:50.511982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:50.511982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:50.512231 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:50.512382 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:50.512646 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 03:22:50.512798 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260927 03:22:50.518800 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abf6bd27e740419c92dd877fec7429ce: Generated new TSK 0
I20260927 03:22:50.519026 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:22:50.546435 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37741
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:50.626997 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:50.627213 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:50.627238 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:50.628383 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:50.628417 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:50.628441 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:22:50.629832 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37741
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:50.630038 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:50.630224 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:50.635141 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:50.635208 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:50.635200 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:50.635317 506 server_base.cc:1034] running on GCE node
I20260927 03:22:50.635529 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:50.635775 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:50.636925 506 hybrid_clock.cc:648] HybridClock initialized: now 1790479370636913 us; error 26 us; skew 500 ppm
Sep 27 03:22:50 dist-test-slave-gw2v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:50.639408 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:22:50.639855 506 webserver.cc:460] Webserver started at http://127.0.98.1:41399/ using document root <none> and password file <none>
I20260927 03:22:50.640048 506 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:50.640103 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:50.640231 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:50.641276 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "4447d564008e4cf5861d96a01007c9f0"
format_stamp: "Formatted at 2026-09-27 03:22:50 on dist-test-slave-gw2v"
server_key: "6353c7f7487b48725088c797bbc69aac"
server_key_iv: "660b3d0cb79b15d91554a55dea56f0c6"
server_key_version: "encryptionkey@0"
I20260927 03:22:50.641636 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4447d564008e4cf5861d96a01007c9f0"
format_stamp: "Formatted at 2026-09-27 03:22:50 on dist-test-slave-gw2v"
server_key: "6353c7f7487b48725088c797bbc69aac"
server_key_iv: "660b3d0cb79b15d91554a55dea56f0c6"
server_key_version: "encryptionkey@0"
I20260927 03:22:50.642906 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:22:50.643678 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:50.643857 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:22:50.643939 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "4447d564008e4cf5861d96a01007c9f0"
format_stamp: "Formatted at 2026-09-27 03:22:50 on dist-test-slave-gw2v"
server_key: "6353c7f7487b48725088c797bbc69aac"
server_key_iv: "660b3d0cb79b15d91554a55dea56f0c6"
server_key_version: "encryptionkey@0"
I20260927 03:22:50.644004 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:50.654359 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:50.656500 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:50.656616 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:50.656909 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:22:50.657196 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:22:50.657229 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:50.657250 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:22:50.657274 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:50.662848 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42531
I20260927 03:22:50.662904 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42531 every 8 connection(s)
I20260927 03:22:50.663193 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:22:50 dist-test-slave-gw2v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:22:50.671382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 506
I20260927 03:22:50.671546 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:22:50.671748 392 external_mini_cluster.cc:1428] Setting key 4979eddd625162587aa2edbd91ecb086
I20260927 03:22:50.672088 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:50.663933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33859 (local address 127.0.98.62:37741)
0927 03:22:50.664001 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:50.664004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:50.664180 (+ 176us) server_negotiation.cc:407] Connection header received
0927 03:22:50.664671 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:50.664672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:50.664684 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:50.664730 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:50.665672 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:50.666185 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:50.667209 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:50.667409 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:50.667465 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:50.669440 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:50.669441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:50.669442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:50.669450 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:50.670660 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:50.671089 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:50.671091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:50.671091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:50.671154 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:50.671422 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:50.671423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:50.671423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:50.671556 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:50.671621 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:50.671896 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 03:22:50.671998 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:22:50.672271 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:50.663870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37741 (local address 127.0.98.1:33859)
0927 03:22:50.664058 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:50.664094 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:50.664574 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:50.664815 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:50.664818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:50.665376 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:50.665570 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:50.665576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:50.666272 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:50.666273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:50.667124 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:50.667126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:50.667477 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:50.667478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:50.667592 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:50.668058 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:50.668066 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:50.669344 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:50.670735 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:50.670741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:50.670743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:50.671007 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:50.671219 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:50.671220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:50.671220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:50.671352 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:50.671620 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:50.671627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:50.671771 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 03:22:50.671864 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 03:22:50.672024 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
I20260927 03:22:50.672896 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37741
I20260927 03:22:50.672989 635 heartbeater.cc:464] Registering TS with master...
I20260927 03:22:50.673202 635 heartbeater.cc:511] Master 127.0.98.62:37741 requested a full tablet report, sending...
I20260927 03:22:50.673746 444 ts_manager.cc:194] Registered new tserver with Master: 4447d564008e4cf5861d96a01007c9f0 (127.0.98.1:42531)
I20260927 03:22:50.674664 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33859
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:22:50.698007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37741
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:50.776822 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:50.777117 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:50.777151 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:50.778309 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:50.778347 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:50.778371 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:22:50.779780 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37741
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:50.779968 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:50.780158 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:22:50.785044 640 server_base.cc:1034] running on GCE node
W20260927 03:22:50.785015 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:50.785010 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:50.785044 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:50.785449 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:50.785686 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:50.786895 640 hybrid_clock.cc:648] HybridClock initialized: now 1790479370786880 us; error 28 us; skew 500 ppm
Sep 27 03:22:50 dist-test-slave-gw2v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:50.789551 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:22:50.790014 640 webserver.cc:460] Webserver started at http://127.0.98.2:38125/ using document root <none> and password file <none>
I20260927 03:22:50.790251 640 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:50.790302 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:50.790442 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:50.791558 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "61f8a08273d74813bd1acf55805108dd"
format_stamp: "Formatted at 2026-09-27 03:22:50 on dist-test-slave-gw2v"
server_key: "07a82cccf2478a5400b5a9e2907a02d6"
server_key_iv: "57de34f8307b7bfdbda8b0e3e7fc94b5"
server_key_version: "encryptionkey@0"
I20260927 03:22:50.791896 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "61f8a08273d74813bd1acf55805108dd"
format_stamp: "Formatted at 2026-09-27 03:22:50 on dist-test-slave-gw2v"
server_key: "07a82cccf2478a5400b5a9e2907a02d6"
server_key_iv: "57de34f8307b7bfdbda8b0e3e7fc94b5"
server_key_version: "encryptionkey@0"
I20260927 03:22:50.793227 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:22:50.793982 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:50.794183 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:22:50.794271 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "61f8a08273d74813bd1acf55805108dd"
format_stamp: "Formatted at 2026-09-27 03:22:50 on dist-test-slave-gw2v"
server_key: "07a82cccf2478a5400b5a9e2907a02d6"
server_key_iv: "57de34f8307b7bfdbda8b0e3e7fc94b5"
server_key_version: "encryptionkey@0"
I20260927 03:22:50.794341 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:50.808579 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:50.810493 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:50.810632 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:50.810890 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:22:50.811208 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:22:50.811241 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:50.811270 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:22:50.811295 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:50.817606 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39711
I20260927 03:22:50.817656 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39711 every 8 connection(s)
I20260927 03:22:50.817977 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:22:50.822649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 640
I20260927 03:22:50.822753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:22:50.822938 392 external_mini_cluster.cc:1428] Setting key 2d8206e6d86da07e2a9f83c8ba5028fc
Sep 27 03:22:50 dist-test-slave-gw2v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:22:50.827288 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:50.818728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34343 (local address 127.0.98.62:37741)
0927 03:22:50.818863 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:50.818865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:50.819011 (+ 146us) server_negotiation.cc:407] Connection header received
0927 03:22:50.819497 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:50.819498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:50.819508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:50.819561 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:50.820486 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:50.820978 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:50.821977 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:50.822167 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:50.822220 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:50.824431 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:50.824433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:50.824433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:50.824441 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:50.825762 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:50.826252 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:50.826254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:50.826254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:50.826314 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:50.826548 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:50.826549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:50.826549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:50.826729 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:50.826790 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:50.827106 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 03:22:50.827195 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 03:22:50.827483 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:50.818688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37741 (local address 127.0.98.2:34343)
0927 03:22:50.818872 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:50.818922 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:50.819412 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:50.819624 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:50.819627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:50.820206 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:50.820411 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:50.820416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:50.821050 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:50.821051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:50.821885 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:50.821887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:50.822229 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:50.822230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:50.822332 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:50.822928 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:50.822937 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:50.824327 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:50.825896 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:50.825903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:50.825905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:50.826169 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:50.826371 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:50.826372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:50.826372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:50.826480 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:50.826791 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:50.826797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:50.826972 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 03:22:50.827065 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 03:22:50.827231 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260927 03:22:50.828343 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37741
I20260927 03:22:50.828466 769 heartbeater.cc:464] Registering TS with master...
I20260927 03:22:50.828754 769 heartbeater.cc:511] Master 127.0.98.62:37741 requested a full tablet report, sending...
I20260927 03:22:50.829185 444 ts_manager.cc:194] Registered new tserver with Master: 61f8a08273d74813bd1acf55805108dd (127.0.98.2:39711)
I20260927 03:22:50.829681 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34343
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:22:50.851570 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37741
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:50.933454 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:50.933677 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:50.933700 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:50.934871 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:50.934917 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:50.934943 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:22:50.936352 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37741
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:50.936542 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:50.936733 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:50.941540 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:50.941602 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:50.941700 774 server_base.cc:1034] running on GCE node
W20260927 03:22:50.941749 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:50.941955 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:50.942179 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:50.943348 774 hybrid_clock.cc:648] HybridClock initialized: now 1790479370943333 us; error 27 us; skew 500 ppm
Sep 27 03:22:50 dist-test-slave-gw2v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:50.945884 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:22:50.946331 774 webserver.cc:460] Webserver started at http://127.0.98.3:35027/ using document root <none> and password file <none>
I20260927 03:22:50.946604 774 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:50.946661 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:50.946763 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:50.947974 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "9cca1e5ca5d34aad83f881e6a7fe0d60"
format_stamp: "Formatted at 2026-09-27 03:22:50 on dist-test-slave-gw2v"
server_key: "957729cd71fce646ecf694a351049d1d"
server_key_iv: "50db2b264245223a46ad3e385e5c1f96"
server_key_version: "encryptionkey@0"
I20260927 03:22:50.948308 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9cca1e5ca5d34aad83f881e6a7fe0d60"
format_stamp: "Formatted at 2026-09-27 03:22:50 on dist-test-slave-gw2v"
server_key: "957729cd71fce646ecf694a351049d1d"
server_key_iv: "50db2b264245223a46ad3e385e5c1f96"
server_key_version: "encryptionkey@0"
I20260927 03:22:50.949693 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:22:50.950874 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:50.951030 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:22:50.951113 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "9cca1e5ca5d34aad83f881e6a7fe0d60"
format_stamp: "Formatted at 2026-09-27 03:22:50 on dist-test-slave-gw2v"
server_key: "957729cd71fce646ecf694a351049d1d"
server_key_iv: "50db2b264245223a46ad3e385e5c1f96"
server_key_version: "encryptionkey@0"
I20260927 03:22:50.951172 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:50.970763 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:50.973030 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:50.973145 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:50.973333 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:22:50.973634 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:22:50.973670 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:50.973690 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:22:50.973701 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:50.979677 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44053
I20260927 03:22:50.979741 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44053 every 8 connection(s)
I20260927 03:22:50.980018 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:22:50 dist-test-slave-gw2v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:22:50.986845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 774
I20260927 03:22:50.986943 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:22:50.987114 392 external_mini_cluster.cc:1428] Setting key bf5d03e75bd6cc6cc6dcbe897b2eb737
I20260927 03:22:50.989226 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:50.980826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60929 (local address 127.0.98.62:37741)
0927 03:22:50.980912 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:50.980914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:50.981041 (+ 127us) server_negotiation.cc:407] Connection header received
0927 03:22:50.981570 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:50.981571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:50.981585 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:50.981632 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:50.982630 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:50.983178 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:50.984140 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:50.984332 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:50.984388 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:50.986397 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:50.986399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:50.986399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:50.986407 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:50.988047 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:50.988481 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:50.988481 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:50.988482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:50.988534 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:50.988725 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:50.988725 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:50.988726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:50.988829 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:50.988878 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:50.989082 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 03:22:50.989165 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 03:22:50.989425 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:50.980701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37741 (local address 127.0.98.3:60929)
0927 03:22:50.980907 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:50.980957 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:50.981471 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:50.981716 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:50.981720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:50.982340 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:50.982534 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:50.982539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:50.983258 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:50.983260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:50.984061 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:50.984063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:50.984399 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:50.984400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:50.984521 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:50.984993 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:50.985002 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:50.986311 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:50.988123 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:50.988129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:50.988131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:50.988416 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:50.988584 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:50.988584 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:50.988585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:50.988678 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:50.988875 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:50.988878 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:50.988990 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 03:22:50.989067 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:22:50.989187 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20260927 03:22:50.989856 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37741
I20260927 03:22:50.989944 903 heartbeater.cc:464] Registering TS with master...
I20260927 03:22:50.990088 903 heartbeater.cc:511] Master 127.0.98.62:37741 requested a full tablet report, sending...
I20260927 03:22:50.990419 444 ts_manager.cc:194] Registered new tserver with Master: 9cca1e5ca5d34aad83f881e6a7fe0d60 (127.0.98.3:44053)
I20260927 03:22:50.990787 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60929
I20260927 03:22:50.999195 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:22:51 dist-test-slave-gw2v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479371, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:22:51 dist-test-slave-gw2v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479371, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:22:51.023365 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:51.015752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39776 (local address 127.0.98.1:42531)
0927 03:22:51.015939 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:51.015945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:51.015955 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:22:51.015980 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:51.015982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:51.015996 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:51.016061 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:51.016592 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:51.017336 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:51.018063 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:51.018391 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:51.018439 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:51.020808 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:51.020818 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:51.020818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:51.020825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:51.022256 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:51.022566 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:51.022569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:51.022569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:51.022635 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:51.022897 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:51.022898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:51.022899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:51.023038 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:51.023150 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:51.023205 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 03:22:51.023284 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":105,"threads_started":1}
W20260927 03:22:51.023716 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39776
Sep 27 03:22:51 dist-test-slave-gw2v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:22:51 dist-test-slave-gw2v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:22:51.045885 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:51.038527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.0.98.1:42531)
0927 03:22:51.038582 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:51.038584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:51.038686 (+ 102us) server_negotiation.cc:407] Connection header received
0927 03:22:51.038752 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:51.038753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:51.038765 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:51.038843 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:51.039403 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:51.039887 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:51.040722 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:51.040915 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:51.040959 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:51.043392 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:51.043394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:51.043394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:51.043404 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:51.044921 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:51.045284 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:51.045285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:51.045286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:51.045340 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:51.045484 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:51.045484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:51.045484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:51.045570 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:51.045656 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:51.045730 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 03:22:51.045818 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:22:51.046419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 506
I20260927 03:22:51.050998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 640
I20260927 03:22:51.055792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 774
I20260927 03:22:51.060050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 412
2026-09-27T03:22:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1840 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:22:51 dist-test-slave-gw2v krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:22:51 dist-test-slave-gw2v krb5kdc[917](info): set up 2 sockets
Sep 27 03:22:51 dist-test-slave-gw2v krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 03:22:52.018203 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:22:52 dist-test-slave-gw2v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:22:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:22:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:22:52.133970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45783
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:52.213380 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:52.213598 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:52.213622 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:52.214798 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:22:52.214864 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:52.214879 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:52.214892 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:22:52.214916 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:22:52.216413 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:52.216647 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:52.216859 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:52.221715 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:52.221805 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:52.221915 930 server_base.cc:1034] running on GCE node
W20260927 03:22:52.221966 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:52.222196 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:52.222507 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:52.223636 930 hybrid_clock.cc:648] HybridClock initialized: now 1790479372223625 us; error 30 us; skew 500 ppm
Sep 27 03:22:52 dist-test-slave-gw2v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:52.226009 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:22:52.226457 930 webserver.cc:460] Webserver started at http://127.0.98.62:46877/ using document root <none> and password file <none>
I20260927 03:22:52.226708 930 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:52.226755 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:52.226904 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:52.227941 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "528fe464c7f94bd4a183b05851091a64"
format_stamp: "Formatted at 2026-09-27 03:22:52 on dist-test-slave-gw2v"
server_key: "9eb7cfeac705ccb8474f5c73543535e8"
server_key_iv: "ca3067cdd2e64124965085c43a834dd6"
server_key_version: "encryptionkey@0"
I20260927 03:22:52.228258 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "528fe464c7f94bd4a183b05851091a64"
format_stamp: "Formatted at 2026-09-27 03:22:52 on dist-test-slave-gw2v"
server_key: "9eb7cfeac705ccb8474f5c73543535e8"
server_key_iv: "ca3067cdd2e64124965085c43a834dd6"
server_key_version: "encryptionkey@0"
I20260927 03:22:52.229537 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:22:52.230221 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:52.230363 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:22:52.230441 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "528fe464c7f94bd4a183b05851091a64"
format_stamp: "Formatted at 2026-09-27 03:22:52 on dist-test-slave-gw2v"
server_key: "9eb7cfeac705ccb8474f5c73543535e8"
server_key_iv: "ca3067cdd2e64124965085c43a834dd6"
server_key_version: "encryptionkey@0"
I20260927 03:22:52.230509 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:52.245116 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:52.247148 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:52.247272 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:52.250975 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45783
I20260927 03:22:52.251021 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45783 every 8 connection(s)
I20260927 03:22:52.251351 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:22:52.252187 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:52.254504 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64: Bootstrap starting.
I20260927 03:22:52.255115 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:52.255366 998 log.cc:824] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:52.256064 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64: No bootstrap required, opened a new log
I20260927 03:22:52.257726 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528fe464c7f94bd4a183b05851091a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45783 } }
I20260927 03:22:52.257874 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:52.257905 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528fe464c7f94bd4a183b05851091a64, State: Initialized, Role: FOLLOWER
I20260927 03:22:52.258049 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528fe464c7f94bd4a183b05851091a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45783 } }
I20260927 03:22:52.258126 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:22:52.258172 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:22:52.258215 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:52.258369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 930
I20260927 03:22:52.258460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:22:52.258618 392 external_mini_cluster.cc:1428] Setting key b49de5c0ed2fe6926d6576597e1f1fc2
I20260927 03:22:52.258872 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528fe464c7f94bd4a183b05851091a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45783 } }
I20260927 03:22:52.258987 998 leader_election.cc:302] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 528fe464c7f94bd4a183b05851091a64; no voters:
I20260927 03:22:52.259158 998 leader_election.cc:288] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:22:52.259418 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:22:52.259565 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:22:52.259691 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [term 1 LEADER]: Becoming Leader. State: Replica: 528fe464c7f94bd4a183b05851091a64, State: Running, Role: LEADER
I20260927 03:22:52.259794 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528fe464c7f94bd4a183b05851091a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45783 } }
I20260927 03:22:52.260290 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "528fe464c7f94bd4a183b05851091a64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528fe464c7f94bd4a183b05851091a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45783 } } }
I20260927 03:22:52.260398 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:52.260296 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 528fe464c7f94bd4a183b05851091a64. Latest consensus state: current_term: 1 leader_uuid: "528fe464c7f94bd4a183b05851091a64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528fe464c7f94bd4a183b05851091a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45783 } } }
I20260927 03:22:52.260763 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64 [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:52.260914 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:22:52.261411 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:22:52.263356 1012 catalog_manager.cc:1347] Generated new cluster ID: 066debddc60e411396037c5d28897cb2
I20260927 03:22:52.263405 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:22:52 dist-test-slave-gw2v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:22:52.270533 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.259129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.0.98.62:45783)
0927 03:22:52.259337 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:52.259347 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:52.259380 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:22:52.259522 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:52.259530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:52.259596 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:52.259746 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:52.260912 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.262313 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.263390 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.264004 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.264114 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.266943 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:52.266948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:52.266948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:52.266962 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:52.268879 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:52.269295 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:52.269298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:52.269298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:52.269402 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:52.269592 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:52.269592 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:52.269592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:52.269884 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:52.270046 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:52.270164 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 03:22:52.270300 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":112,"threads_started":1}
I20260927 03:22:52.278366 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:22:52.278972 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:22:52.285740 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 528fe464c7f94bd4a183b05851091a64: Generated new TSK 0
I20260927 03:22:52.285948 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:22:52.314110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45783
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:52.394541 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:52.394780 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:52.394814 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:52.396034 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:52.396076 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:52.396111 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:22:52.397570 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45783
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:52.397816 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:52.398026 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:52.402966 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:52.403018 1024 server_base.cc:1034] running on GCE node
W20260927 03:22:52.403151 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:52.403028 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:52.403417 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:52.403626 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:52.404767 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790479372404754 us; error 28 us; skew 500 ppm
Sep 27 03:22:52 dist-test-slave-gw2v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:52.407212 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:22:52.407663 1024 webserver.cc:460] Webserver started at http://127.0.98.1:37425/ using document root <none> and password file <none>
I20260927 03:22:52.407864 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:52.407913 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:52.408031 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:52.409122 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "d7d19c1ecefe4ec3b775fb1cc07ed120"
format_stamp: "Formatted at 2026-09-27 03:22:52 on dist-test-slave-gw2v"
server_key: "2a85e667ea04f265c31909e1f21ae23a"
server_key_iv: "c74dae161c2872be5e1b1d01c06706a3"
server_key_version: "encryptionkey@0"
I20260927 03:22:52.409448 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d7d19c1ecefe4ec3b775fb1cc07ed120"
format_stamp: "Formatted at 2026-09-27 03:22:52 on dist-test-slave-gw2v"
server_key: "2a85e667ea04f265c31909e1f21ae23a"
server_key_iv: "c74dae161c2872be5e1b1d01c06706a3"
server_key_version: "encryptionkey@0"
I20260927 03:22:52.410732 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:22:52.411440 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:52.411628 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:52.411731 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "d7d19c1ecefe4ec3b775fb1cc07ed120"
format_stamp: "Formatted at 2026-09-27 03:22:52 on dist-test-slave-gw2v"
server_key: "2a85e667ea04f265c31909e1f21ae23a"
server_key_iv: "c74dae161c2872be5e1b1d01c06706a3"
server_key_version: "encryptionkey@0"
I20260927 03:22:52.411792 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:52.418079 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:52.419938 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:52.420066 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:52.420293 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:22:52.420578 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:22:52.420609 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:52.420645 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:22:52.420670 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:52.426301 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33199
I20260927 03:22:52.426348 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33199 every 8 connection(s)
I20260927 03:22:52.426678 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:22:52.429164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 1024
I20260927 03:22:52.429255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:22:52.429400 392 external_mini_cluster.cc:1428] Setting key 00afcc4dc02ed84fe93323cbd830c810
Sep 27 03:22:52 dist-test-slave-gw2v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:22:52.436672 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.427416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44247 (local address 127.0.98.62:45783)
0927 03:22:52.427481 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:52.427483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:52.427700 (+ 217us) server_negotiation.cc:407] Connection header received
0927 03:22:52.428148 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:52.428149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:52.428161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:52.428210 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:52.429161 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.429824 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.431066 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.431274 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.431330 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.433721 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:52.433723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:52.433723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:52.433734 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:52.435332 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:52.435727 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:52.435728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:52.435729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:52.435791 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:52.436000 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:52.436001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:52.436001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:52.436147 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:52.436211 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:52.436477 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 03:22:52.436574 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:22:52.436748 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.427398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45783 (local address 127.0.98.1:44247)
0927 03:22:52.427573 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:52.427609 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:52.428072 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:52.428267 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:52.428270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:52.428825 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:52.429057 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.429064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.429911 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.429912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.430977 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.430981 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.431344 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.431345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.431484 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.432003 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:52.432011 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:52.433579 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:52.435411 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:52.435417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:52.435419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:52.435653 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:52.435856 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:52.435856 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:52.435857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:52.435949 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:52.436207 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:52.436210 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:52.436328 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 03:22:52.436436 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 03:22:52.436566 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
I20260927 03:22:52.437222 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45783
I20260927 03:22:52.437314 1153 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 03:22:52.437498 1153 heartbeater.cc:511] Master 127.0.98.62:45783 requested a full tablet report, sending...
I20260927 03:22:52.438020 962 ts_manager.cc:194] Registered new tserver with Master: d7d19c1ecefe4ec3b775fb1cc07ed120 (127.0.98.1:33199)
I20260927 03:22:52.438614 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:22:52.454588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45783
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:52.534147 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:52.534363 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:52.534386 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:52.535640 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:52.535688 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:52.535714 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:22:52.537139 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45783
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:52.537352 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:52.537585 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:52.542661 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:52.542661 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:52.542685 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:52.542690 1158 server_base.cc:1034] running on GCE node
I20260927 03:22:52.543090 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:52.543330 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:52.544534 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790479372544520 us; error 31 us; skew 500 ppm
Sep 27 03:22:52 dist-test-slave-gw2v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:52.547091 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:22:52.547550 1158 webserver.cc:460] Webserver started at http://127.0.98.2:41547/ using document root <none> and password file <none>
I20260927 03:22:52.547816 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:52.547866 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:52.547986 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:52.548992 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "e2c97a9897c14575b241e4698c9700a1"
format_stamp: "Formatted at 2026-09-27 03:22:52 on dist-test-slave-gw2v"
server_key: "d3d1f0b47c479963abe91166f893741c"
server_key_iv: "81d04ca81368baa4bad478e5f01cb7ff"
server_key_version: "encryptionkey@0"
I20260927 03:22:52.549305 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e2c97a9897c14575b241e4698c9700a1"
format_stamp: "Formatted at 2026-09-27 03:22:52 on dist-test-slave-gw2v"
server_key: "d3d1f0b47c479963abe91166f893741c"
server_key_iv: "81d04ca81368baa4bad478e5f01cb7ff"
server_key_version: "encryptionkey@0"
I20260927 03:22:52.550478 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:22:52.551234 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:52.551381 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:52.551460 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "e2c97a9897c14575b241e4698c9700a1"
format_stamp: "Formatted at 2026-09-27 03:22:52 on dist-test-slave-gw2v"
server_key: "d3d1f0b47c479963abe91166f893741c"
server_key_iv: "81d04ca81368baa4bad478e5f01cb7ff"
server_key_version: "encryptionkey@0"
I20260927 03:22:52.551517 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:52.565670 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:52.567663 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:52.567795 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:52.568006 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:22:52.568290 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:22:52.568322 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:52.568351 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:22:52.568368 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:52.573856 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44907
I20260927 03:22:52.573896 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44907 every 8 connection(s)
I20260927 03:22:52.574234 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:22:52.578995 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 1158
I20260927 03:22:52.579082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:22:52.579247 392 external_mini_cluster.cc:1428] Setting key f9fbda9e566db34981c33b4cd2b95e36
Sep 27 03:22:52 dist-test-slave-gw2v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:22:52.583513 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.574997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45989 (local address 127.0.98.62:45783)
0927 03:22:52.575040 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:52.575042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:52.575245 (+ 203us) server_negotiation.cc:407] Connection header received
0927 03:22:52.575738 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:52.575739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:52.575750 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:52.575805 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:52.576689 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.577213 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.578181 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.578379 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.578453 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.580765 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:52.580767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:52.580767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:52.580773 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:52.582226 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:52.582576 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:52.582577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:52.582578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:52.582637 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:52.582873 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:52.582874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:52.582874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:52.582988 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:52.583044 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:52.583321 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 03:22:52.583418 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:22:52.583580 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.574979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45783 (local address 127.0.98.2:45989)
0927 03:22:52.575128 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:52.575162 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:52.575621 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:52.575863 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:52.575867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:52.576399 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:52.576598 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.576603 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.577276 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.577277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.578085 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.578087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.578471 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.578472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.578570 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.579130 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:52.579141 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:52.580668 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:52.582296 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:52.582301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:52.582303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:52.582498 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:52.582694 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:52.582695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:52.582695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:52.582799 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:52.583040 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:52.583046 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:52.583176 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 03:22:52.583274 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 03:22:52.583416 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260927 03:22:52.584038 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45783
I20260927 03:22:52.584123 1287 heartbeater.cc:464] Registering TS with master...
I20260927 03:22:52.584288 1287 heartbeater.cc:511] Master 127.0.98.62:45783 requested a full tablet report, sending...
I20260927 03:22:52.584652 962 ts_manager.cc:194] Registered new tserver with Master: e2c97a9897c14575b241e4698c9700a1 (127.0.98.2:44907)
I20260927 03:22:52.585136 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45989
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:22:52.605360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45783
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:52.686138 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:52.686345 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:52.686367 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:52.687506 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:52.687539 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:52.687564 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:22:52.688905 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45783
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:52.689085 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:52.689262 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:52.694293 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:52.694279 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:52.694331 1292 server_base.cc:1034] running on GCE node
W20260927 03:22:52.694283 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:52.694734 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:52.695004 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:52.696158 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790479372696145 us; error 30 us; skew 500 ppm
Sep 27 03:22:52 dist-test-slave-gw2v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:52.698552 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:22:52.699033 1292 webserver.cc:460] Webserver started at http://127.0.98.3:38613/ using document root <none> and password file <none>
I20260927 03:22:52.699227 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:52.699265 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:52.699354 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:52.700335 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "2a87e8d8832f41d28423fd9e8c211fee"
format_stamp: "Formatted at 2026-09-27 03:22:52 on dist-test-slave-gw2v"
server_key: "a54a8fee89798cab10288124284650fd"
server_key_iv: "7d7adc1887551f2aab58e53308ee51d1"
server_key_version: "encryptionkey@0"
I20260927 03:22:52.700609 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2a87e8d8832f41d28423fd9e8c211fee"
format_stamp: "Formatted at 2026-09-27 03:22:52 on dist-test-slave-gw2v"
server_key: "a54a8fee89798cab10288124284650fd"
server_key_iv: "7d7adc1887551f2aab58e53308ee51d1"
server_key_version: "encryptionkey@0"
I20260927 03:22:52.701735 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:22:52.702390 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:52.702574 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:22:52.702651 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "2a87e8d8832f41d28423fd9e8c211fee"
format_stamp: "Formatted at 2026-09-27 03:22:52 on dist-test-slave-gw2v"
server_key: "a54a8fee89798cab10288124284650fd"
server_key_iv: "7d7adc1887551f2aab58e53308ee51d1"
server_key_version: "encryptionkey@0"
I20260927 03:22:52.702718 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:52.719461 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:52.721295 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:52.721432 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:52.721642 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:22:52.721932 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:22:52.721964 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:52.721992 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:22:52.722010 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:52.727448 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36989
I20260927 03:22:52.727507 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36989 every 8 connection(s)
I20260927 03:22:52.727787 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:22:52.729910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 1292
I20260927 03:22:52.729984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:22:52.730123 392 external_mini_cluster.cc:1428] Setting key 8f60a5c4a353a6813a02ab0e026c7ad7
Sep 27 03:22:52 dist-test-slave-gw2v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:22:52.736562 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.728445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48999 (local address 127.0.98.62:45783)
0927 03:22:52.728520 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:52.728523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:52.728619 (+ 96us) server_negotiation.cc:407] Connection header received
0927 03:22:52.729134 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:52.729135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:52.729145 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:52.729196 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:52.730111 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.730670 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.731750 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.731937 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.732029 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.734126 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:52.734127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:52.734127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:52.734134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:52.735333 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:52.735725 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:52.735726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:52.735726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:52.735775 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:52.735957 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:52.735958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:52.735958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:52.736083 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:52.736131 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:52.736392 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 03:22:52.736469 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 03:22:52.736680 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.728368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45783 (local address 127.0.98.3:48999)
0927 03:22:52.728519 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:52.728558 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:52.729041 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:52.729256 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:52.729259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:52.729812 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:52.730036 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.730042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.730762 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.730763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.731672 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.731675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.732003 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.732004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.732134 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.732644 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:52.732651 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:52.734057 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:52.735399 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:52.735404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:52.735406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:52.735665 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:52.735825 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:52.735825 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:52.735826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:52.735917 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:52.736127 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:52.736130 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:52.736259 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 03:22:52.736337 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 03:22:52.736454 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260927 03:22:52.737144 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45783
I20260927 03:22:52.737231 1421 heartbeater.cc:464] Registering TS with master...
I20260927 03:22:52.737375 1421 heartbeater.cc:511] Master 127.0.98.62:45783 requested a full tablet report, sending...
I20260927 03:22:52.737705 962 ts_manager.cc:194] Registered new tserver with Master: 2a87e8d8832f41d28423fd9e8c211fee (127.0.98.3:36989)
I20260927 03:22:52.738054 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48999
I20260927 03:22:52.741573 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:22:52.748855 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.743092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.0.98.62:45783)
0927 03:22:52.743145 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:52.743147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:52.743242 (+ 95us) server_negotiation.cc:407] Connection header received
0927 03:22:52.743304 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:52.743305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:52.743314 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:52.743353 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:52.743933 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.744418 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.745106 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.745298 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.745350 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.746457 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:52.746459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:52.746459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:52.746466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:52.747863 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:52.748163 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:52.748164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:52.748164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:52.748215 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:52.748373 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:52.748373 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:52.748373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:52.748510 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:52.748595 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:52.748697 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 03:22:52.748767 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:22:52.751135 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:22:52.752749 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:22:52.759087 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.754934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.0.98.2:44907)
0927 03:22:52.755235 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:52.755239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:52.755264 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:22:52.755350 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:52.755352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:52.755367 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:52.755439 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:22:52.755689 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.756818 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.758039 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.758642 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.758712 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.758769 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:52.758922 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 03:22:52.758970 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
I20260927 03:22:52.759099 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.754850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44907 (local address 127.0.0.1:54840)
0927 03:22:52.755170 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:52.755210 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:52.755264 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:52.755563 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:52.755565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:52.755567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:22:52.755627 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.755629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.756923 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.756924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.757962 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.757965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.758714 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.758715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.758789 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.758805 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 03:22:52.758907 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 03:22:52.758957 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":39,"thread_start_us":73,"threads_started":1}
I20260927 03:22:52.759812 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.754847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36989 (local address 127.0.0.1:44186)
0927 03:22:52.755169 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:52.755210 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:52.755296 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:52.756065 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:52.756066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:52.756068 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:22:52.756111 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.756113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.757485 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.757486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.758527 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.758529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.759588 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.759589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.759650 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.759654 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:22:52.759705 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:22:52.759743 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":79,"threads_started":1}
I20260927 03:22:52.759753 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 507cd037605e44e1b5acd7a52662e044 (DEFAULT_TABLE table=test-table [id=b96f0ed365214d99aa974fdc9b297048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:22:52.759832 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.754951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.0.98.3:36989)
0927 03:22:52.755159 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:52.755164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:52.755269 (+ 105us) server_negotiation.cc:407] Connection header received
0927 03:22:52.755376 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:52.755378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:52.755395 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:52.755960 (+ 565us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:22:52.756181 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.757370 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.758599 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.759521 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.759581 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.759637 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:52.759719 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 03:22:52.759754 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":87,"threads_started":1}
I20260927 03:22:52.760113 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507cd037605e44e1b5acd7a52662e044. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:52.762605 1435 tablet_bootstrap.cc:492] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1: Bootstrap starting.
I20260927 03:22:52.763197 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.755055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33199 (local address 127.0.0.1:54736)
0927 03:22:52.755372 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:52.755381 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:52.755942 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:52.756115 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:52.756116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:52.756117 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:22:52.756163 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.756164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.757141 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.757142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.758324 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.758327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.763000 (+ 4673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.763001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.763062 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.763067 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:22:52.763111 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:22:52.763148 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":63,"threads_started":1}
I20260927 03:22:52.763304 1435 tablet_bootstrap.cc:654] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:52.763334 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 507cd037605e44e1b5acd7a52662e044 (DEFAULT_TABLE table=test-table [id=b96f0ed365214d99aa974fdc9b297048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:22:52.763463 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.755245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.0.98.1:33199)
0927 03:22:52.755425 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:52.755443 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:52.756001 (+ 558us) server_negotiation.cc:407] Connection header received
0927 03:22:52.756039 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:52.756041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:52.756052 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:52.756108 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:22:52.756213 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.756926 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.758409 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.762841 (+ 4432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.762905 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.762969 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:52.763332 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 03:22:52.763377 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260927 03:22:52.763725 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507cd037605e44e1b5acd7a52662e044. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:52.763813 1435 log.cc:824] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:52.764021 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 507cd037605e44e1b5acd7a52662e044 (DEFAULT_TABLE table=test-table [id=b96f0ed365214d99aa974fdc9b297048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:22:52.764338 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507cd037605e44e1b5acd7a52662e044. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:52.764556 1435 tablet_bootstrap.cc:492] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1: No bootstrap required, opened a new log
I20260927 03:22:52.764621 1435 ts_tablet_manager.cc:1397] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:22:52.766132 1437 tablet_bootstrap.cc:492] T 507cd037605e44e1b5acd7a52662e044 P 2a87e8d8832f41d28423fd9e8c211fee: Bootstrap starting.
I20260927 03:22:52.766417 1435 raft_consensus.cc:348] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c97a9897c14575b241e4698c9700a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44907 } } peers { permanent_uuid: "2a87e8d8832f41d28423fd9e8c211fee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "d7d19c1ecefe4ec3b775fb1cc07ed120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } }
I20260927 03:22:52.766605 1435 raft_consensus.cc:374] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:52.766638 1435 raft_consensus.cc:729] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c97a9897c14575b241e4698c9700a1, State: Initialized, Role: FOLLOWER
I20260927 03:22:52.766713 1437 tablet_bootstrap.cc:654] T 507cd037605e44e1b5acd7a52662e044 P 2a87e8d8832f41d28423fd9e8c211fee: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:52.766747 1435 consensus_queue.cc:260] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c97a9897c14575b241e4698c9700a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44907 } } peers { permanent_uuid: "2a87e8d8832f41d28423fd9e8c211fee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "d7d19c1ecefe4ec3b775fb1cc07ed120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } }
I20260927 03:22:52.767035 1435 ts_tablet_manager.cc:1428] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260927 03:22:52.767046 1287 heartbeater.cc:503] Master 127.0.98.62:45783 was elected leader, sending a full tablet report...
I20260927 03:22:52.767567 1437 log.cc:824] T 507cd037605e44e1b5acd7a52662e044 P 2a87e8d8832f41d28423fd9e8c211fee: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:52.767632 1439 tablet_bootstrap.cc:492] T 507cd037605e44e1b5acd7a52662e044 P d7d19c1ecefe4ec3b775fb1cc07ed120: Bootstrap starting.
I20260927 03:22:52.768203 1439 tablet_bootstrap.cc:654] T 507cd037605e44e1b5acd7a52662e044 P d7d19c1ecefe4ec3b775fb1cc07ed120: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:52.768277 1437 tablet_bootstrap.cc:492] T 507cd037605e44e1b5acd7a52662e044 P 2a87e8d8832f41d28423fd9e8c211fee: No bootstrap required, opened a new log
I20260927 03:22:52.768342 1437 ts_tablet_manager.cc:1397] T 507cd037605e44e1b5acd7a52662e044 P 2a87e8d8832f41d28423fd9e8c211fee: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:22:52.769208 1439 log.cc:824] T 507cd037605e44e1b5acd7a52662e044 P d7d19c1ecefe4ec3b775fb1cc07ed120: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:52.769908 1439 tablet_bootstrap.cc:492] T 507cd037605e44e1b5acd7a52662e044 P d7d19c1ecefe4ec3b775fb1cc07ed120: No bootstrap required, opened a new log
I20260927 03:22:52.769975 1439 ts_tablet_manager.cc:1397] T 507cd037605e44e1b5acd7a52662e044 P d7d19c1ecefe4ec3b775fb1cc07ed120: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:22:52.770303 1437 raft_consensus.cc:348] T 507cd037605e44e1b5acd7a52662e044 P 2a87e8d8832f41d28423fd9e8c211fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c97a9897c14575b241e4698c9700a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44907 } } peers { permanent_uuid: "2a87e8d8832f41d28423fd9e8c211fee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "d7d19c1ecefe4ec3b775fb1cc07ed120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } }
I20260927 03:22:52.770473 1437 raft_consensus.cc:374] T 507cd037605e44e1b5acd7a52662e044 P 2a87e8d8832f41d28423fd9e8c211fee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:52.770512 1437 raft_consensus.cc:729] T 507cd037605e44e1b5acd7a52662e044 P 2a87e8d8832f41d28423fd9e8c211fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a87e8d8832f41d28423fd9e8c211fee, State: Initialized, Role: FOLLOWER
I20260927 03:22:52.770619 1437 consensus_queue.cc:260] T 507cd037605e44e1b5acd7a52662e044 P 2a87e8d8832f41d28423fd9e8c211fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c97a9897c14575b241e4698c9700a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44907 } } peers { permanent_uuid: "2a87e8d8832f41d28423fd9e8c211fee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "d7d19c1ecefe4ec3b775fb1cc07ed120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } }
I20260927 03:22:52.770893 1437 ts_tablet_manager.cc:1428] T 507cd037605e44e1b5acd7a52662e044 P 2a87e8d8832f41d28423fd9e8c211fee: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 03:22:52.770897 1421 heartbeater.cc:503] Master 127.0.98.62:45783 was elected leader, sending a full tablet report...
I20260927 03:22:52.771950 1439 raft_consensus.cc:348] T 507cd037605e44e1b5acd7a52662e044 P d7d19c1ecefe4ec3b775fb1cc07ed120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c97a9897c14575b241e4698c9700a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44907 } } peers { permanent_uuid: "2a87e8d8832f41d28423fd9e8c211fee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "d7d19c1ecefe4ec3b775fb1cc07ed120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } }
I20260927 03:22:52.772078 1439 raft_consensus.cc:374] T 507cd037605e44e1b5acd7a52662e044 P d7d19c1ecefe4ec3b775fb1cc07ed120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:52.772097 1439 raft_consensus.cc:729] T 507cd037605e44e1b5acd7a52662e044 P d7d19c1ecefe4ec3b775fb1cc07ed120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d19c1ecefe4ec3b775fb1cc07ed120, State: Initialized, Role: FOLLOWER
I20260927 03:22:52.772177 1439 consensus_queue.cc:260] T 507cd037605e44e1b5acd7a52662e044 P d7d19c1ecefe4ec3b775fb1cc07ed120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c97a9897c14575b241e4698c9700a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44907 } } peers { permanent_uuid: "2a87e8d8832f41d28423fd9e8c211fee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "d7d19c1ecefe4ec3b775fb1cc07ed120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } }
I20260927 03:22:52.772379 1439 ts_tablet_manager.cc:1428] T 507cd037605e44e1b5acd7a52662e044 P d7d19c1ecefe4ec3b775fb1cc07ed120: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:22:52.772423 1153 heartbeater.cc:503] Master 127.0.98.62:45783 was elected leader, sending a full tablet report...
W20260927 03:22:52.824803 1288 tablet.cc:2369] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:22:52.914619 1438 raft_consensus.cc:482] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:22:52.914744 1438 raft_consensus.cc:504] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c97a9897c14575b241e4698c9700a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44907 } } peers { permanent_uuid: "2a87e8d8832f41d28423fd9e8c211fee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "d7d19c1ecefe4ec3b775fb1cc07ed120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } }
I20260927 03:22:52.915112 1438 leader_election.cc:288] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a87e8d8832f41d28423fd9e8c211fee (127.0.98.3:36989), d7d19c1ecefe4ec3b775fb1cc07ed120 (127.0.98.1:33199)
I20260927 03:22:52.918075 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.915327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36989 (local address 127.0.98.2:48143)
0927 03:22:52.915406 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:52.915420 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:52.915489 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:52.915672 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:52.915673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:52.915676 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:22:52.915742 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.915744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.916439 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.916441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.917246 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.917248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.917834 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.917835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.917910 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.917915 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:22:52.917964 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:22:52.918009 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 03:22:52.918107 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.915408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48143 (local address 127.0.98.3:36989)
0927 03:22:52.915476 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:52.915479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:52.915490 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:22:52.915551 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:52.915552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:52.915567 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:52.915639 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:22:52.915818 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.916368 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.917332 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.917766 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.917819 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.917828 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:52.917974 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 03:22:52.918017 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 03:22:52.918193 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.915464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33199 (local address 127.0.98.2:36233)
0927 03:22:52.915729 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:52.915739 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:52.915790 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:52.915908 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:52.915909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:52.915910 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:22:52.915971 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.915973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.916580 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.916581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.917340 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:52.917342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.917940 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.917941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:52.918013 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.918018 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:22:52.918071 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:22:52.918123 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":187,"threads_started":1}
I20260927 03:22:52.918201 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.915631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36233 (local address 127.0.98.1:33199)
0927 03:22:52.915674 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:52.915676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:52.915798 (+ 122us) server_negotiation.cc:407] Connection header received
0927 03:22:52.915830 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:52.915831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:52.915842 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:52.915885 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:22:52.916037 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.916499 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.917431 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.917853 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.917921 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.917931 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:52.918085 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 03:22:52.918141 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:22:52.918505 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507cd037605e44e1b5acd7a52662e044" candidate_uuid: "e2c97a9897c14575b241e4698c9700a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a87e8d8832f41d28423fd9e8c211fee" is_pre_election: true
I20260927 03:22:52.918514 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507cd037605e44e1b5acd7a52662e044" candidate_uuid: "e2c97a9897c14575b241e4698c9700a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d19c1ecefe4ec3b775fb1cc07ed120" is_pre_election: true
I20260927 03:22:52.918675 1375 raft_consensus.cc:2454] T 507cd037605e44e1b5acd7a52662e044 P 2a87e8d8832f41d28423fd9e8c211fee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c97a9897c14575b241e4698c9700a1 in term 0.
I20260927 03:22:52.918675 1107 raft_consensus.cc:2454] T 507cd037605e44e1b5acd7a52662e044 P d7d19c1ecefe4ec3b775fb1cc07ed120 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c97a9897c14575b241e4698c9700a1 in term 0.
I20260927 03:22:52.918923 1176 leader_election.cc:302] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7d19c1ecefe4ec3b775fb1cc07ed120, e2c97a9897c14575b241e4698c9700a1; no voters:
I20260927 03:22:52.919064 1438 raft_consensus.cc:2781] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:22:52.919130 1438 raft_consensus.cc:482] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:22:52.919147 1438 raft_consensus.cc:3037] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [term 0 FOLLOWER]: Advancing to term 1
W20260927 03:22:52.927417 1154 tablet.cc:2369] T 507cd037605e44e1b5acd7a52662e044 P d7d19c1ecefe4ec3b775fb1cc07ed120: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:22:52.941324 1438 raft_consensus.cc:504] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c97a9897c14575b241e4698c9700a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44907 } } peers { permanent_uuid: "2a87e8d8832f41d28423fd9e8c211fee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "d7d19c1ecefe4ec3b775fb1cc07ed120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } }
I20260927 03:22:52.941581 1438 leader_election.cc:288] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [CANDIDATE]: Term 1 election: Requested vote from peers 2a87e8d8832f41d28423fd9e8c211fee (127.0.98.3:36989), d7d19c1ecefe4ec3b775fb1cc07ed120 (127.0.98.1:33199)
I20260927 03:22:52.941826 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507cd037605e44e1b5acd7a52662e044" candidate_uuid: "e2c97a9897c14575b241e4698c9700a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a87e8d8832f41d28423fd9e8c211fee"
I20260927 03:22:52.941876 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507cd037605e44e1b5acd7a52662e044" candidate_uuid: "e2c97a9897c14575b241e4698c9700a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d19c1ecefe4ec3b775fb1cc07ed120"
I20260927 03:22:52.941979 1107 raft_consensus.cc:3037] T 507cd037605e44e1b5acd7a52662e044 P d7d19c1ecefe4ec3b775fb1cc07ed120 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:52.941979 1375 raft_consensus.cc:3037] T 507cd037605e44e1b5acd7a52662e044 P 2a87e8d8832f41d28423fd9e8c211fee [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:52.942934 1107 raft_consensus.cc:2454] T 507cd037605e44e1b5acd7a52662e044 P d7d19c1ecefe4ec3b775fb1cc07ed120 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c97a9897c14575b241e4698c9700a1 in term 1.
I20260927 03:22:52.942934 1375 raft_consensus.cc:2454] T 507cd037605e44e1b5acd7a52662e044 P 2a87e8d8832f41d28423fd9e8c211fee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c97a9897c14575b241e4698c9700a1 in term 1.
I20260927 03:22:52.943158 1176 leader_election.cc:302] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7d19c1ecefe4ec3b775fb1cc07ed120, e2c97a9897c14575b241e4698c9700a1; no voters:
I20260927 03:22:52.943264 1438 raft_consensus.cc:2781] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:22:52.943447 1438 raft_consensus.cc:686] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [term 1 LEADER]: Becoming Leader. State: Replica: e2c97a9897c14575b241e4698c9700a1, State: Running, Role: LEADER
I20260927 03:22:52.943554 1438 consensus_queue.cc:237] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c97a9897c14575b241e4698c9700a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44907 } } peers { permanent_uuid: "2a87e8d8832f41d28423fd9e8c211fee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } } peers { permanent_uuid: "d7d19c1ecefe4ec3b775fb1cc07ed120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } }
I20260927 03:22:52.944458 961 catalog_manager.cc:5495] T 507cd037605e44e1b5acd7a52662e044 P e2c97a9897c14575b241e4698c9700a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c97a9897c14575b241e4698c9700a1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e2c97a9897c14575b241e4698c9700a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c97a9897c14575b241e4698c9700a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a87e8d8832f41d28423fd9e8c211fee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d19c1ecefe4ec3b775fb1cc07ed120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33199 } health_report { overall_health: UNKNOWN } } }
W20260927 03:22:52.978289 1422 tablet.cc:2369] T 507cd037605e44e1b5acd7a52662e044 P 2a87e8d8832f41d28423fd9e8c211fee: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 03:22:52 dist-test-slave-gw2v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:22:52.989872 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:52.982411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.0.98.1:33199)
0927 03:22:52.982541 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:52.982543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:52.982552 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:22:52.982575 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:52.982576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:52.982587 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:52.982638 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:52.983209 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.983709 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.984499 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:52.984700 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:52.984760 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:52.987064 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:52.987076 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:52.987076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:52.987083 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:52.988633 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:52.989021 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:52.989025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:52.989025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:52.989098 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:52.989262 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:52.989263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:52.989263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:52.989402 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:52.989518 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:52.989724 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 03:22:52.989794 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
Sep 27 03:22:52 dist-test-slave-gw2v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479372, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:22:53 dist-test-slave-gw2v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479372, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:22:53.044994 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:53.037105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.0.98.1:33199)
0927 03:22:53.037176 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:53.037178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:53.037253 (+ 75us) server_negotiation.cc:407] Connection header received
0927 03:22:53.037322 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:53.037323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:53.037332 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:53.037383 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:53.037978 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:53.038482 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:53.039236 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:53.039461 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:53.039519 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:53.042289 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:53.042291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:53.042291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:53.042301 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:53.043857 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:53.044219 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:53.044219 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:53.044220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:53.044269 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:53.044452 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:53.044453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:53.044453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:53.044552 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:53.044622 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:53.044857 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 03:22:53.044924 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 03:22:53 dist-test-slave-gw2v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:22:53 dist-test-slave-gw2v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:22:53.143359 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:53.136200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.0.98.1:33199)
0927 03:22:53.136251 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:53.136254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:53.136371 (+ 117us) server_negotiation.cc:407] Connection header received
0927 03:22:53.136514 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:53.136515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:53.136528 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:53.136577 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:53.137191 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:53.137664 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:53.138400 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:53.138585 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:53.138641 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:53.141000 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:53.141001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:53.141002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:53.141008 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:53.142282 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:53.142613 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:53.142615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:53.142615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:53.142670 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:53.142852 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:53.142853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:53.142853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:53.142949 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:53.143034 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:53.143196 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 03:22:53.143267 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 03:22:53.143800 1067 tablet_service.cc:2908] Rejecting scan request for tablet 507cd037605e44e1b5acd7a52662e044: Uninitialized: clean time has not yet been initialized
I20260927 03:22:53.144615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 1024
I20260927 03:22:53.149675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 1158
I20260927 03:22:53.153723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 1292
I20260927 03:22:53.157346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 930
2026-09-27T03:22:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2098 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:22:54 dist-test-slave-gw2v krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:22:54 dist-test-slave-gw2v krb5kdc[1459](info): set up 2 sockets
Sep 27 03:22:54 dist-test-slave-gw2v krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260927 03:22:55.675778 392 mini_kdc.cc:121] Time spent starting KDC: real 2.498s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 03:22:55.801368 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.125s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/test-user.keytab.
W20260927 03:22:56.408890 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.438s user 0.001s sys 0.001s
Sep 27 03:22:56 dist-test-slave-gw2v krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:22:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:22:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260927 03:22:57.210577 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.781s user 0.000s sys 0.002s
I20260927 03:22:57.210963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40669
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:57.294874 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:57.295143 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:57.295167 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:57.296357 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:22:57.296399 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 03:22:57.296412 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:57.296424 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:57.296435 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:22:57.296445 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:22:57.298049 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40669
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:57.298288 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:57.298509 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:57.303308 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:57.303333 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:57.303560 1472 server_base.cc:1034] running on GCE node
W20260927 03:22:57.303431 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:57.303889 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:57.304152 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:57.305300 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790479377305303 us; error 43 us; skew 500 ppm
Sep 27 03:22:57 dist-test-slave-gw2v krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:57.308058 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:22:57.308516 1472 webserver.cc:460] Webserver started at http://127.0.98.62:36047/ using document root <none> and password file <none>
I20260927 03:22:57.308725 1472 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:57.308773 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:57.308898 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:57.310031 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "8c6bc9bdc18d4520b425f4fcaf834a31"
format_stamp: "Formatted at 2026-09-27 03:22:57 on dist-test-slave-gw2v"
server_key: "d8e32c45c24bb65bdd1cf1130f32ac6c"
server_key_iv: "9b801beb59271e087fc8cc70e0725cc5"
server_key_version: "encryptionkey@0"
I20260927 03:22:57.310369 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c6bc9bdc18d4520b425f4fcaf834a31"
format_stamp: "Formatted at 2026-09-27 03:22:57 on dist-test-slave-gw2v"
server_key: "d8e32c45c24bb65bdd1cf1130f32ac6c"
server_key_iv: "9b801beb59271e087fc8cc70e0725cc5"
server_key_version: "encryptionkey@0"
I20260927 03:22:57.311728 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:22:57.312430 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:57.312597 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:22:57.312711 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "8c6bc9bdc18d4520b425f4fcaf834a31"
format_stamp: "Formatted at 2026-09-27 03:22:57 on dist-test-slave-gw2v"
server_key: "d8e32c45c24bb65bdd1cf1130f32ac6c"
server_key_iv: "9b801beb59271e087fc8cc70e0725cc5"
server_key_version: "encryptionkey@0"
I20260927 03:22:57.312778 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:57.324049 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:57.326035 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:57.326172 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:57.330765 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40669
I20260927 03:22:57.330864 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40669 every 8 connection(s)
I20260927 03:22:57.331305 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:22:57.332129 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:57.334403 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31: Bootstrap starting.
I20260927 03:22:57.335043 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:57.335323 1550 log.cc:824] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:57.335633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 1472
I20260927 03:22:57.335711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:22:57.335868 392 external_mini_cluster.cc:1428] Setting key f2c9066fe8619c71f736db3925188646
I20260927 03:22:57.336074 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31: No bootstrap required, opened a new log
I20260927 03:22:57.338268 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6bc9bdc18d4520b425f4fcaf834a31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40669 } }
I20260927 03:22:57.338418 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:57.338451 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c6bc9bdc18d4520b425f4fcaf834a31, State: Initialized, Role: FOLLOWER
I20260927 03:22:57.338644 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6bc9bdc18d4520b425f4fcaf834a31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40669 } }
I20260927 03:22:57.338737 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:22:57.338768 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:22:57.338840 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 03:22:57 dist-test-slave-gw2v krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:22:57.346400 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:57.336436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47962 (local address 127.0.98.62:40669)
0927 03:22:57.336595 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:57.336605 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:57.336638 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:22:57.338242 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:57.338250 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:57.338308 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:57.338441 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:57.339215 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:57.340125 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:57.340824 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:57.341280 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:57.341361 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:57.343629 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:57.343633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:57.343633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:57.343643 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:57.345168 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:57.345466 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:57.345468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:57.345468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:57.345544 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:57.345706 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:57.345707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:57.345707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:57.345923 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:57.346015 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:57.346141 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 03:22:57.346243 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260927 03:22:57.360983 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6bc9bdc18d4520b425f4fcaf834a31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40669 } }
I20260927 03:22:57.361169 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c6bc9bdc18d4520b425f4fcaf834a31; no voters:
I20260927 03:22:57.361392 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:22:57.361544 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:22:57.361738 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [term 1 LEADER]: Becoming Leader. State: Replica: 8c6bc9bdc18d4520b425f4fcaf834a31, State: Running, Role: LEADER
I20260927 03:22:57.361728 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:22:57.361925 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6bc9bdc18d4520b425f4fcaf834a31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40669 } }
I20260927 03:22:57.362433 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c6bc9bdc18d4520b425f4fcaf834a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6bc9bdc18d4520b425f4fcaf834a31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40669 } } }
I20260927 03:22:57.362524 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:57.362747 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c6bc9bdc18d4520b425f4fcaf834a31. Latest consensus state: current_term: 1 leader_uuid: "8c6bc9bdc18d4520b425f4fcaf834a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6bc9bdc18d4520b425f4fcaf834a31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40669 } } }
I20260927 03:22:57.362880 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31 [sys.catalog]: This master's current role is: LEADER
I20260927 03:22:57.363114 1563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:22:57.363680 1563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:22:57.366546 1563 catalog_manager.cc:1347] Generated new cluster ID: d518558143944400b96b89a946c67d2d
I20260927 03:22:57.366600 1563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:22:57.386142 1563 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:22:57.386940 1563 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:22:57.402962 1563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31: Generated new TSK 0
I20260927 03:22:57.403198 1563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260927 03:22:58.551255 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 1.146s user 0.001s sys 0.002s
I20260927 03:22:58.551633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40669
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:58.633363 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:58.633602 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:58.633630 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:58.634759 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:58.634807 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:58.634852 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:22:58.634872 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:22:58.636413 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:58.636646 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:58.636896 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:58.642226 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:58.642266 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:58.642232 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:58.642731 1576 server_base.cc:1034] running on GCE node
I20260927 03:22:58.643016 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:58.643263 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:58.644418 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790479378644394 us; error 59 us; skew 500 ppm
Sep 27 03:22:58 dist-test-slave-gw2v krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:58.647184 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:22:58.647691 1576 webserver.cc:460] Webserver started at http://127.0.98.1:40477/ using document root <none> and password file <none>
I20260927 03:22:58.647869 1576 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:58.647908 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:58.648000 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:58.649019 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "1ade04976c334268b09b674a3c9197c3"
format_stamp: "Formatted at 2026-09-27 03:22:58 on dist-test-slave-gw2v"
server_key: "f3bc397cd8f8f99d29635864b3d3130d"
server_key_iv: "de948d1b6b810b20e030ef9d044577bf"
server_key_version: "encryptionkey@0"
I20260927 03:22:58.649313 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1ade04976c334268b09b674a3c9197c3"
format_stamp: "Formatted at 2026-09-27 03:22:58 on dist-test-slave-gw2v"
server_key: "f3bc397cd8f8f99d29635864b3d3130d"
server_key_iv: "de948d1b6b810b20e030ef9d044577bf"
server_key_version: "encryptionkey@0"
I20260927 03:22:58.650556 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:22:58.651345 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:58.651523 1576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:22:58.651607 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "1ade04976c334268b09b674a3c9197c3"
format_stamp: "Formatted at 2026-09-27 03:22:58 on dist-test-slave-gw2v"
server_key: "f3bc397cd8f8f99d29635864b3d3130d"
server_key_iv: "de948d1b6b810b20e030ef9d044577bf"
server_key_version: "encryptionkey@0"
I20260927 03:22:58.651660 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:58.679436 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:58.681640 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:58.681763 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:58.682317 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:22:58.682371 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:58.682402 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:22:58.682417 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:58.688791 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36215
I20260927 03:22:58.688974 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36215 every 8 connection(s)
I20260927 03:22:58.689486 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:22:58 dist-test-slave-gw2v krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:22:58.693127 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:58.682811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44711 (local address 127.0.98.62:40669)
0927 03:22:58.682942 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:58.682945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:58.683153 (+ 208us) server_negotiation.cc:407] Connection header received
0927 03:22:58.683731 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:58.683733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:58.683749 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:58.683832 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:58.684734 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.685523 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.687075 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.687390 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.687484 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:58.690202 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:58.690204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:58.690204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:58.690212 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:58.691574 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:58.692053 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:58.692054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:58.692055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:58.692123 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:58.692444 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:58.692445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:58.692445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:58.692616 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:58.692685 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:58.692962 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 03:22:58.693050 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":53,"threads_started":1}
I20260927 03:22:58.693302 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:58.682859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40669 (local address 127.0.98.1:44711)
0927 03:22:58.683030 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:58.683068 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:58.683594 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:58.683910 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:58.683915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:58.684284 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:58.684615 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:58.684623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.685622 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.685623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:58.686976 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:58.686980 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.687494 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.687495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:58.687649 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:58.688275 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:58.688286 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:58.690099 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:58.691670 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:58.691676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:58.691679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:58.691954 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:58.692217 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:58.692218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:58.692218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:58.692343 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:58.692691 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:58.692698 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:58.692840 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 03:22:58.692941 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 03:22:58.693087 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260927 03:22:58.693833 1708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40669
I20260927 03:22:58.693995 1708 heartbeater.cc:464] Registering TS with master...
I20260927 03:22:58.694193 1708 heartbeater.cc:511] Master 127.0.98.62:40669 requested a full tablet report, sending...
I20260927 03:22:58.694809 1503 ts_manager.cc:194] Registered new tserver with Master: 1ade04976c334268b09b674a3c9197c3 (127.0.98.1:36215)
I20260927 03:22:58.695739 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44711
I20260927 03:22:58.697669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 1576
I20260927 03:22:58.697767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:22:58.697996 392 external_mini_cluster.cc:1428] Setting key d9961356f2d2d3b70349724e99f93927
I20260927 03:22:58.701004 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:58.695110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36479 (local address 127.0.98.62:40669)
0927 03:22:58.695165 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:58.695167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:58.695297 (+ 130us) server_negotiation.cc:407] Connection header received
0927 03:22:58.695351 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:58.695352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:58.695361 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:58.695401 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:58.695741 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.696331 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.697291 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.697504 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.697581 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:58.698635 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:58.698636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:58.698637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:58.698644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:58.699979 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:58.700291 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:58.700292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:58.700293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:58.700344 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:58.700514 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:58.700515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:58.700515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:58.700623 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:58.700676 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:58.700813 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 03:22:58.700897 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:22:58.701004 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:58.695038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40669 (local address 127.0.98.1:36479)
0927 03:22:58.695227 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:58.695238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:58.695300 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:58.695420 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:58.695421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:58.695614 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:58.695678 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:58.695680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.696411 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.696412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:58.697172 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:58.697175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.697597 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.697598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:58.697664 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:58.698152 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:58.698158 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:58.698557 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:58.700057 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:58.700058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:58.700059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:58.700228 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:58.700397 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:58.700398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:58.700398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:58.700465 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:58.700675 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:58.700676 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:58.700726 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 03:22:58.700785 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 03:22:58.700897 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":46,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:22:58.727113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40669
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:58.807660 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:58.807876 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:58.807899 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:58.809068 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:58.809113 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:58.809132 1717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:22:58.809149 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:22:58.810650 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:58.810913 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:58.811188 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:58.816061 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:58.816107 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:58.816071 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:58.816417 1717 server_base.cc:1034] running on GCE node
I20260927 03:22:58.816605 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:58.816846 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:58.817994 1717 hybrid_clock.cc:648] HybridClock initialized: now 1790479378817980 us; error 29 us; skew 500 ppm
Sep 27 03:22:58 dist-test-slave-gw2v krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:58.820508 1717 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:22:58.820992 1717 webserver.cc:460] Webserver started at http://127.0.98.2:40391/ using document root <none> and password file <none>
I20260927 03:22:58.821197 1717 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:58.821245 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:58.821370 1717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:58.822436 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "a57801eab5e049a4a7a83d318b93d535"
format_stamp: "Formatted at 2026-09-27 03:22:58 on dist-test-slave-gw2v"
server_key: "fc62005b70545feecfd081e19f439a12"
server_key_iv: "087a156254961d0c011b40f2332255d7"
server_key_version: "encryptionkey@0"
I20260927 03:22:58.822762 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a57801eab5e049a4a7a83d318b93d535"
format_stamp: "Formatted at 2026-09-27 03:22:58 on dist-test-slave-gw2v"
server_key: "fc62005b70545feecfd081e19f439a12"
server_key_iv: "087a156254961d0c011b40f2332255d7"
server_key_version: "encryptionkey@0"
I20260927 03:22:58.824098 1717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:22:58.824898 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:58.825071 1717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:22:58.825153 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "a57801eab5e049a4a7a83d318b93d535"
format_stamp: "Formatted at 2026-09-27 03:22:58 on dist-test-slave-gw2v"
server_key: "fc62005b70545feecfd081e19f439a12"
server_key_iv: "087a156254961d0c011b40f2332255d7"
server_key_version: "encryptionkey@0"
I20260927 03:22:58.825217 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:58.837054 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:58.839303 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:58.839445 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:58.839958 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:22:58.840001 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:58.840035 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:22:58.840071 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:58.846511 1717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45259
I20260927 03:22:58.846568 1847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45259 every 8 connection(s)
Sep 27 03:22:58 dist-test-slave-gw2v krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:22:58.847000 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:22:58.850262 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:58.840470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39463 (local address 127.0.98.62:40669)
0927 03:22:58.840534 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:58.840536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:58.840789 (+ 253us) server_negotiation.cc:407] Connection header received
0927 03:22:58.841348 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:58.841359 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:58.841373 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:58.841449 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:58.842301 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.843126 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.844463 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.844754 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.844835 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:58.847280 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:58.847282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:58.847282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:58.847292 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:58.848690 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:58.849202 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:58.849204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:58.849204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:58.849266 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:58.849581 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:58.849582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:58.849583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:58.849712 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:58.849775 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:58.850116 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 03:22:58.850194 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:22:58.850524 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:58.840465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40669 (local address 127.0.98.2:39463)
0927 03:22:58.840685 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:58.840715 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:58.841230 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:58.841523 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:58.841528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:58.841898 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:58.842193 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:58.842201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.843243 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.843245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:58.844364 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:58.844367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.844836 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.844837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:58.844947 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:58.845539 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:58.845549 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:58.847154 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:58.848775 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:58.848781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:58.848784 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:58.849119 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:58.849351 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:58.849352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:58.849353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:58.849494 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:58.849803 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:58.849810 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:58.849976 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 03:22:58.850088 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 03:22:58.850244 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
I20260927 03:22:58.851090 1848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40669
I20260927 03:22:58.851222 1848 heartbeater.cc:464] Registering TS with master...
I20260927 03:22:58.851428 1848 heartbeater.cc:511] Master 127.0.98.62:40669 requested a full tablet report, sending...
I20260927 03:22:58.851991 1504 ts_manager.cc:194] Registered new tserver with Master: a57801eab5e049a4a7a83d318b93d535 (127.0.98.2:45259)
I20260927 03:22:58.852062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 1717
I20260927 03:22:58.852136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:22:58.852301 392 external_mini_cluster.cc:1428] Setting key d6482a715a7e75c4e5faabcbb569b038
I20260927 03:22:58.852481 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39463
I20260927 03:22:58.858686 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:58.852652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53519 (local address 127.0.98.62:40669)
0927 03:22:58.852697 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:58.852699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:58.852707 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:22:58.852728 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:58.852729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:58.852740 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:58.852807 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:58.853200 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.853758 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.854623 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.854935 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.855035 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:58.856108 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:58.856111 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:58.856112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:58.856121 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:58.857477 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:58.857904 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:58.857905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:58.857906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:58.857995 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:58.858191 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:58.858192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:58.858192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:58.858341 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:58.858391 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:58.858551 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 03:22:58.858620 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:22:58.858807 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:58.852200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40669 (local address 127.0.98.2:53519)
0927 03:22:58.852330 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:58.852342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:58.852390 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:58.852819 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:58.852820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:58.853040 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:58.853108 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:58.853111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.853870 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.853872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:58.854551 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:58.854553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.855034 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.855035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:58.855085 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:58.855574 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:58.855579 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:58.855954 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:58.857570 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:58.857571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:58.857572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:58.857793 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:58.858066 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:58.858067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:58.858067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:58.858133 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:58.858408 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:58.858409 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:58.858458 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 03:22:58.858530 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 03:22:58.858638 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:22:58.878320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40669
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:22:58.960995 1857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:22:58.961227 1857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:22:58.961252 1857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:22:58.962532 1857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:22:58.962584 1857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:22:58.962605 1857 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:22:58.962628 1857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:22:58.964273 1857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:22:58.964529 1857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:22:58.964722 1857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:22:58.969926 1865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:58.970151 1862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:22:58.970162 1863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:22:58.970587 1857 server_base.cc:1034] running on GCE node
I20260927 03:22:58.970788 1857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:22:58.971083 1857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:22:58.972244 1857 hybrid_clock.cc:648] HybridClock initialized: now 1790479378972231 us; error 39 us; skew 500 ppm
Sep 27 03:22:58 dist-test-slave-gw2v krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:22:58.974798 1857 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:22:58.975356 1857 webserver.cc:460] Webserver started at http://127.0.98.3:43141/ using document root <none> and password file <none>
I20260927 03:22:58.975561 1857 fs_manager.cc:359] Metadata directory not provided
I20260927 03:22:58.975602 1857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:22:58.975697 1857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:22:58.976792 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "6129c2ee946643c685c3697f6301f646"
format_stamp: "Formatted at 2026-09-27 03:22:58 on dist-test-slave-gw2v"
server_key: "5915b63b8b9d6a5b753a30658f68ae84"
server_key_iv: "7945c037ccc31a52602a744f88ef3429"
server_key_version: "encryptionkey@0"
I20260927 03:22:58.977131 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6129c2ee946643c685c3697f6301f646"
format_stamp: "Formatted at 2026-09-27 03:22:58 on dist-test-slave-gw2v"
server_key: "5915b63b8b9d6a5b753a30658f68ae84"
server_key_iv: "7945c037ccc31a52602a744f88ef3429"
server_key_version: "encryptionkey@0"
I20260927 03:22:58.978404 1857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:22:58.979228 1872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:58.979413 1857 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:22:58.979490 1857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "6129c2ee946643c685c3697f6301f646"
format_stamp: "Formatted at 2026-09-27 03:22:58 on dist-test-slave-gw2v"
server_key: "5915b63b8b9d6a5b753a30658f68ae84"
server_key_iv: "7945c037ccc31a52602a744f88ef3429"
server_key_version: "encryptionkey@0"
I20260927 03:22:58.979563 1857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:22:58.992358 1857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:22:58.994686 1857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:22:58.994848 1857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:22:58.995369 1857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:22:58.995416 1857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:58.995446 1857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:22:58.995464 1857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:59.001555 1857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33459
I20260927 03:22:59.001749 1987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33459 every 8 connection(s)
I20260927 03:22:59.001999 1857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:22:58 dist-test-slave-gw2v krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:22:59.003358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 1857
I20260927 03:22:59.003458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:22:59.003626 392 external_mini_cluster.cc:1428] Setting key 733f9c11a1b740715f101a4fa54284ae
I20260927 03:22:59.005810 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:58.995849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38247 (local address 127.0.98.62:40669)
0927 03:22:58.995930 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:58.995932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:58.996122 (+ 190us) server_negotiation.cc:407] Connection header received
0927 03:22:58.996683 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:58.996684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:58.996698 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:58.996762 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:58.997661 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.998442 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.999999 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.000293 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.000371 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.002911 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:59.002913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:59.002913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:59.002921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:59.004475 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:59.004870 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:59.004872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:59.004872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:59.004955 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:59.005194 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:59.005195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:59.005195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:59.005312 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:59.005367 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.005635 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.005721 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 03:22:59.006000 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:58.995845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40669 (local address 127.0.98.3:38247)
0927 03:22:58.995994 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:58.996033 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:58.996563 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:58.996848 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:58.996852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:58.997212 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:58.997530 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:58.997538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:58.998558 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:58.998559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:58.999904 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:58.999907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.000387 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.000388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.000545 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.001149 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:59.001158 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:59.002769 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:59.004549 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:59.004556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:59.004559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:59.004786 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:59.005012 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:59.005013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:59.005013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:59.005130 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:59.005365 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:59.005371 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:59.005509 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 03:22:59.005605 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 03:22:59.005761 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260927 03:22:59.006408 1988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40669
I20260927 03:22:59.006495 1988 heartbeater.cc:464] Registering TS with master...
I20260927 03:22:59.006678 1988 heartbeater.cc:511] Master 127.0.98.62:40669 requested a full tablet report, sending...
I20260927 03:22:59.007155 1503 ts_manager.cc:194] Registered new tserver with Master: 6129c2ee946643c685c3697f6301f646 (127.0.98.3:33459)
I20260927 03:22:59.007660 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38247
I20260927 03:22:59.012706 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.007568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47659 (local address 127.0.98.62:40669)
0927 03:22:59.007605 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.007606 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.007703 (+ 97us) server_negotiation.cc:407] Connection header received
0927 03:22:59.007826 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.007826 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.007836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.007885 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:59.008211 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.008685 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.009352 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.009547 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.009599 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.010450 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:59.010451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:59.010452 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:59.010459 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:59.011720 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:59.012035 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:59.012036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:59.012037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:59.012088 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:59.012243 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:59.012243 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:59.012244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:59.012352 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:59.012404 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.012536 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.012609 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:22:59.012722 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.007512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40669 (local address 127.0.98.3:47659)
0927 03:22:59.007653 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:59.007663 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:59.007752 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:59.007890 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:59.007890 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:59.008071 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:59.008128 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.008129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.008769 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.008770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.009276 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.009278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.009603 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.009604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.009668 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.010085 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:59.010090 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:59.010372 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:59.011810 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:59.011811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:59.011811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:59.011968 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:59.012138 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:59.012139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:59.012139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:59.012194 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:59.012403 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:59.012404 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:59.012446 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 03:22:59.012512 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:22:59.012611 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260927 03:22:59.015486 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:22:59 dist-test-slave-gw2v krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479379, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:22:59 dist-test-slave-gw2v krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479379, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:22:59.038955 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.031042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47976 (local address 127.0.98.62:40669)
0927 03:22:59.031147 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.031149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.031157 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:22:59.031182 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.031183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.031192 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.031247 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:59.031954 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.032471 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.033324 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.033538 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.033608 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.036141 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:59.036143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:59.036143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:59.036151 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:59.037640 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:59.038029 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:59.038030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:59.038030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:59.038086 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:59.038282 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:59.038283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:59.038283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:59.038389 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:59.038479 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.038663 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.038747 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 03:22:59.040618 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:22:59.042186 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:22:59.048730 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.044607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33459 (local address 127.0.0.1:56970)
0927 03:22:59.045131 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:59.045140 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:59.045174 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:59.045440 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:59.045441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:59.045444 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:22:59.045492 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.045494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.046357 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.046359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.047529 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.047532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.048423 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.048424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.048505 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.048522 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 03:22:59.048589 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:22:59.048639 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":112,"threads_started":1}
I20260927 03:22:59.048760 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.044684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56970 (local address 127.0.98.3:33459)
0927 03:22:59.045276 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.045280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.045290 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:22:59.045326 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.045328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.045343 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.045561 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:22:59.045618 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.046279 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.047600 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.048342 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.048418 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.048500 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.048615 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.048655 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":69,"threads_started":1}
I20260927 03:22:59.048923 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.044601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45259 (local address 127.0.0.1:52638)
0927 03:22:59.044808 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:59.044848 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:59.044925 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:59.045494 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:59.045495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:59.045497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:22:59.045531 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.045531 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.046864 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.046865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.047810 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.047812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.048707 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.048708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.048775 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.048780 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:22:59.048808 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 03:22:59.048851 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":101,"threads_started":1}
I20260927 03:22:59.049356 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.045085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.0.98.2:45259)
0927 03:22:59.045334 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.045338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.045348 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:22:59.045382 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.045384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.045399 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.045593 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:22:59.045652 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.046752 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.047886 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.048621 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.049094 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.049187 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.049214 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.049254 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":89,"threads_started":1}
I20260927 03:22:59.049525 1922 tablet_service.cc:1467] Processing CreateTablet for tablet c570a58e86ae4effb343c352570a2f03 (DEFAULT_TABLE table=test-table [id=b34a817e43f543d2b08ccebb1d535040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:22:59.049940 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c570a58e86ae4effb343c352570a2f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:59.050086 1782 tablet_service.cc:1467] Processing CreateTablet for tablet c570a58e86ae4effb343c352570a2f03 (DEFAULT_TABLE table=test-table [id=b34a817e43f543d2b08ccebb1d535040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:22:59.050426 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.044601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36215 (local address 127.0.0.1:52016)
0927 03:22:59.045026 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:59.045035 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:59.045070 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:59.045306 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:59.045310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:59.045323 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:22:59.045399 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.045401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.047876 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.047877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.048711 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.048714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.049840 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.049841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.049909 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.049914 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:22:59.049977 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:22:59.050353 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":4,"thread_start_us":95,"threads_started":1}
I20260927 03:22:59.050463 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c570a58e86ae4effb343c352570a2f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:59.050240 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.044893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52016 (local address 127.0.98.1:36215)
0927 03:22:59.045068 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.045072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.045083 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:22:59.045117 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.045119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.045132 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.045210 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:22:59.045486 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.046753 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.049163 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.049771 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.049838 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.049922 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.050094 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.050138 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260927 03:22:59.051185 1642 tablet_service.cc:1467] Processing CreateTablet for tablet c570a58e86ae4effb343c352570a2f03 (DEFAULT_TABLE table=test-table [id=b34a817e43f543d2b08ccebb1d535040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:22:59.051491 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c570a58e86ae4effb343c352570a2f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:59.053849 2012 tablet_bootstrap.cc:492] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646: Bootstrap starting.
I20260927 03:22:59.054572 2014 tablet_bootstrap.cc:492] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3: Bootstrap starting.
I20260927 03:22:59.054603 2012 tablet_bootstrap.cc:654] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:59.054891 2013 tablet_bootstrap.cc:492] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535: Bootstrap starting.
I20260927 03:22:59.055127 2012 log.cc:824] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:59.055243 2014 tablet_bootstrap.cc:654] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:59.055501 2013 tablet_bootstrap.cc:654] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:59.055542 2014 log.cc:824] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:59.055773 2013 log.cc:824] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535: Log is configured to *not* fsync() on all Append() calls
I20260927 03:22:59.056378 2014 tablet_bootstrap.cc:492] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3: No bootstrap required, opened a new log
I20260927 03:22:59.056455 2014 ts_tablet_manager.cc:1397] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:22:59.056486 2012 tablet_bootstrap.cc:492] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646: No bootstrap required, opened a new log
I20260927 03:22:59.056546 2012 ts_tablet_manager.cc:1397] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:22:59.058475 2012 raft_consensus.cc:348] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.058475 2014 raft_consensus.cc:348] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.058657 2012 raft_consensus.cc:374] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:59.058657 2014 raft_consensus.cc:374] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:59.058700 2012 raft_consensus.cc:729] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6129c2ee946643c685c3697f6301f646, State: Initialized, Role: FOLLOWER
I20260927 03:22:59.058700 2014 raft_consensus.cc:729] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade04976c334268b09b674a3c9197c3, State: Initialized, Role: FOLLOWER
I20260927 03:22:59.058851 2012 consensus_queue.cc:260] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.058851 2014 consensus_queue.cc:260] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.059124 2014 ts_tablet_manager.cc:1428] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:22:59.059124 2012 ts_tablet_manager.cc:1428] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:22:59.059252 1988 heartbeater.cc:503] Master 127.0.98.62:40669 was elected leader, sending a full tablet report...
I20260927 03:22:59.059266 2013 tablet_bootstrap.cc:492] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535: No bootstrap required, opened a new log
I20260927 03:22:59.059276 1708 heartbeater.cc:503] Master 127.0.98.62:40669 was elected leader, sending a full tablet report...
I20260927 03:22:59.059329 2013 ts_tablet_manager.cc:1397] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 03:22:59.061177 2013 raft_consensus.cc:348] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.061313 2013 raft_consensus.cc:374] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:59.061354 2013 raft_consensus.cc:729] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57801eab5e049a4a7a83d318b93d535, State: Initialized, Role: FOLLOWER
I20260927 03:22:59.061472 2013 consensus_queue.cc:260] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.061726 2013 ts_tablet_manager.cc:1428] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:22:59.061779 1848 heartbeater.cc:503] Master 127.0.98.62:40669 was elected leader, sending a full tablet report...
W20260927 03:22:59.097996 1849 tablet.cc:2369] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:22:59.155247 2020 raft_consensus.cc:482] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:22:59.155409 2020 raft_consensus.cc:504] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.155767 2020 leader_election.cc:288] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade04976c334268b09b674a3c9197c3 (127.0.98.1:36215), 6129c2ee946643c685c3697f6301f646 (127.0.98.3:33459)
I20260927 03:22:59.159888 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.156014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52601 (local address 127.0.98.1:36215)
0927 03:22:59.156079 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.156081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.156143 (+ 62us) server_negotiation.cc:407] Connection header received
0927 03:22:59.156218 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.156219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.156231 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.156291 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:22:59.156569 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.157299 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.158870 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.159515 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.159595 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.159607 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.159775 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.159814 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:22:59.159940 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.156088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40411 (local address 127.0.98.3:33459)
0927 03:22:59.156125 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.156127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.156174 (+ 47us) server_negotiation.cc:407] Connection header received
0927 03:22:59.156228 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.156229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.156239 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.156296 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:22:59.156535 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.157302 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.158937 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.159516 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.159595 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.159607 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.159775 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.159880 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:22:59.159982 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.155951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36215 (local address 127.0.98.2:52601)
0927 03:22:59.156081 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:59.156093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:59.156160 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:59.156309 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:59.156309 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:59.156310 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:22:59.156413 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.156415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.157412 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.157413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.158735 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.158738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.159607 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.159608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.159700 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.159705 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:22:59.159765 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:22:59.159813 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":79}
I20260927 03:22:59.160001 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.155964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33459 (local address 127.0.98.2:40411)
0927 03:22:59.156126 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:59.156134 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:59.156183 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:59.156305 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:59.156306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:59.156307 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:22:59.156445 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.156446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.157412 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.157413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.158734 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.158738 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.159607 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.159608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.159700 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.159705 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:22:59.159765 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:22:59.159813 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":94,"thread_start_us":92,"threads_started":1}
I20260927 03:22:59.160338 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570a58e86ae4effb343c352570a2f03" candidate_uuid: "a57801eab5e049a4a7a83d318b93d535" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6129c2ee946643c685c3697f6301f646" is_pre_election: true
I20260927 03:22:59.160481 1942 raft_consensus.cc:2454] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a57801eab5e049a4a7a83d318b93d535 in term 0.
I20260927 03:22:59.160662 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570a58e86ae4effb343c352570a2f03" candidate_uuid: "a57801eab5e049a4a7a83d318b93d535" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade04976c334268b09b674a3c9197c3" is_pre_election: true
I20260927 03:22:59.160785 1662 raft_consensus.cc:2454] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a57801eab5e049a4a7a83d318b93d535 in term 0.
I20260927 03:22:59.161007 1733 leader_election.cc:302] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6129c2ee946643c685c3697f6301f646, a57801eab5e049a4a7a83d318b93d535; no voters:
I20260927 03:22:59.161142 2020 raft_consensus.cc:2781] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:22:59.161198 2020 raft_consensus.cc:482] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:22:59.161221 2020 raft_consensus.cc:3037] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:59.161957 2020 raft_consensus.cc:504] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.162082 2020 leader_election.cc:288] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [CANDIDATE]: Term 1 election: Requested vote from peers 1ade04976c334268b09b674a3c9197c3 (127.0.98.1:36215), 6129c2ee946643c685c3697f6301f646 (127.0.98.3:33459)
I20260927 03:22:59.162259 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570a58e86ae4effb343c352570a2f03" candidate_uuid: "a57801eab5e049a4a7a83d318b93d535" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade04976c334268b09b674a3c9197c3"
I20260927 03:22:59.162322 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570a58e86ae4effb343c352570a2f03" candidate_uuid: "a57801eab5e049a4a7a83d318b93d535" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6129c2ee946643c685c3697f6301f646"
I20260927 03:22:59.162361 1662 raft_consensus.cc:3037] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:59.162417 1942 raft_consensus.cc:3037] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:59.163313 1942 raft_consensus.cc:2454] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a57801eab5e049a4a7a83d318b93d535 in term 1.
I20260927 03:22:59.163331 1662 raft_consensus.cc:2454] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a57801eab5e049a4a7a83d318b93d535 in term 1.
I20260927 03:22:59.163486 1733 leader_election.cc:302] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6129c2ee946643c685c3697f6301f646, a57801eab5e049a4a7a83d318b93d535; no voters:
I20260927 03:22:59.163587 2020 raft_consensus.cc:2781] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:22:59.163774 2020 raft_consensus.cc:686] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [term 1 LEADER]: Becoming Leader. State: Replica: a57801eab5e049a4a7a83d318b93d535, State: Running, Role: LEADER
I20260927 03:22:59.163877 2020 consensus_queue.cc:237] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.164808 1504 catalog_manager.cc:5495] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 reported cstate change: term changed from 0 to 1, leader changed from <none> to a57801eab5e049a4a7a83d318b93d535 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a57801eab5e049a4a7a83d318b93d535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } health_report { overall_health: UNKNOWN } } }
W20260927 03:22:59.190363 1709 tablet.cc:2369] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:22:59.196781 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.194348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52640 (local address 127.0.98.2:45259)
0927 03:22:59.194423 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.194425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.194433 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:22:59.194495 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.194496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.194513 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.194567 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:22:59.194725 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.195197 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.195965 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.196155 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.196207 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.196509 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:22:59.196604 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:22:59.196659 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.196692 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.196709 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 03:22:59.198563 1662 raft_consensus.cc:1261] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3 [term 1 FOLLOWER]: Refusing update from remote peer a57801eab5e049a4a7a83d318b93d535: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:22:59.198563 1942 raft_consensus.cc:1261] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646 [term 1 FOLLOWER]: Refusing update from remote peer a57801eab5e049a4a7a83d318b93d535: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:22:59.198938 2022 consensus_queue.cc:1035] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:22:59.199086 2020 consensus_queue.cc:1035] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:22:59.201574 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333803537194741760 to 7333803537055944704. Current Snapshot: MvccSnapshot[applied={T|T < 7333803537194741760}]
I20260927 03:22:59.201644 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333803537194741760 to 7333803537055944704. Current Snapshot: MvccSnapshot[applied={T|T < 7333803537194741760}]
I20260927 03:22:59.205116 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:22:59.205240 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:22:59.206013 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31: Sending DeleteTablet for 3 replicas of tablet c570a58e86ae4effb343c352570a2f03
I20260927 03:22:59.206348 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet c570a58e86ae4effb343c352570a2f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:22:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52016
I20260927 03:22:59.206359 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet c570a58e86ae4effb343c352570a2f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:22:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52638
I20260927 03:22:59.206429 1922 tablet_service.cc:1514] Processing DeleteTablet for tablet c570a58e86ae4effb343c352570a2f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:22:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56970
I20260927 03:22:59.206621 2037 tablet_replica.cc:330] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646: stopping tablet replica
I20260927 03:22:59.206724 2037 raft_consensus.cc:2229] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:22:59.206897 2037 raft_consensus.cc:2258] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:22:59.206876 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:22:59.207024 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:22:59.207420 2037 ts_tablet_manager.cc:1905] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:22:59.207449 2035 tablet_replica.cc:330] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3: stopping tablet replica
I20260927 03:22:59.207546 2035 raft_consensus.cc:2229] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:22:59.207639 2035 raft_consensus.cc:2258] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:22:59.208359 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 5e25394c0e6049c6a014ebf8125dbc18 (DEFAULT_TABLE table=test-table [id=ddbe3de93c724184a04dd359643d5855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:22:59.208413 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 5e25394c0e6049c6a014ebf8125dbc18 (DEFAULT_TABLE table=test-table [id=ddbe3de93c724184a04dd359643d5855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:22:59.208477 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e25394c0e6049c6a014ebf8125dbc18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:59.208455 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 5e25394c0e6049c6a014ebf8125dbc18 (DEFAULT_TABLE table=test-table [id=ddbe3de93c724184a04dd359643d5855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:22:59.208524 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e25394c0e6049c6a014ebf8125dbc18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:59.208550 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e25394c0e6049c6a014ebf8125dbc18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:59.209766 2013 tablet_bootstrap.cc:492] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535: Bootstrap starting.
I20260927 03:22:59.209791 2012 tablet_bootstrap.cc:492] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646: Bootstrap starting.
I20260927 03:22:59.209817 2014 tablet_bootstrap.cc:492] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3: Bootstrap starting.
I20260927 03:22:59.210155 2013 tablet_bootstrap.cc:654] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:59.210172 2012 tablet_bootstrap.cc:654] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:59.210172 2014 tablet_bootstrap.cc:654] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:59.210956 2014 tablet_bootstrap.cc:492] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3: No bootstrap required, opened a new log
I20260927 03:22:59.210968 2012 tablet_bootstrap.cc:492] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646: No bootstrap required, opened a new log
I20260927 03:22:59.211009 2014 ts_tablet_manager.cc:1397] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:22:59.211009 2012 ts_tablet_manager.cc:1397] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:22:59.211150 2014 raft_consensus.cc:348] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.211150 2012 raft_consensus.cc:348] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.211220 2014 raft_consensus.cc:374] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:59.211220 2012 raft_consensus.cc:374] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:59.211239 2012 raft_consensus.cc:729] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6129c2ee946643c685c3697f6301f646, State: Initialized, Role: FOLLOWER
I20260927 03:22:59.211239 2014 raft_consensus.cc:729] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade04976c334268b09b674a3c9197c3, State: Initialized, Role: FOLLOWER
I20260927 03:22:59.211280 2014 consensus_queue.cc:260] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.211282 2012 consensus_queue.cc:260] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.211370 2012 ts_tablet_manager.cc:1428] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:22:59.211375 2014 ts_tablet_manager.cc:1428] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:59.212283 2035 ts_tablet_manager.cc:1905] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:22:59.213208 2037 ts_tablet_manager.cc:1918] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:22:59.213292 2037 log.cc:1196] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/c570a58e86ae4effb343c352570a2f03
I20260927 03:22:59.213447 2036 tablet_replica.cc:330] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535: stopping tablet replica
I20260927 03:22:59.213552 2037 ts_tablet_manager.cc:1939] T c570a58e86ae4effb343c352570a2f03 P 6129c2ee946643c685c3697f6301f646: Deleting consensus metadata
I20260927 03:22:59.213557 2036 raft_consensus.cc:2229] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:22:59.213696 2036 raft_consensus.cc:2258] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:22:59.213951 1488 catalog_manager.cc:4841] TS 6129c2ee946643c685c3697f6301f646 (127.0.98.3:33459): tablet c570a58e86ae4effb343c352570a2f03 (table test-table [id=b34a817e43f543d2b08ccebb1d535040]) successfully deleted
I20260927 03:22:59.214390 2035 ts_tablet_manager.cc:1918] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:22:59.214468 2035 log.cc:1196] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/wal/wals/c570a58e86ae4effb343c352570a2f03
I20260927 03:22:59.214545 2013 tablet_bootstrap.cc:492] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535: No bootstrap required, opened a new log
I20260927 03:22:59.214592 2013 ts_tablet_manager.cc:1397] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 03:22:59.214706 2035 ts_tablet_manager.cc:1939] T c570a58e86ae4effb343c352570a2f03 P 1ade04976c334268b09b674a3c9197c3: Deleting consensus metadata
I20260927 03:22:59.214772 2013 raft_consensus.cc:348] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.214879 2013 raft_consensus.cc:374] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:59.214908 2013 raft_consensus.cc:729] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57801eab5e049a4a7a83d318b93d535, State: Initialized, Role: FOLLOWER
I20260927 03:22:59.214950 2013 consensus_queue.cc:260] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.215044 2013 ts_tablet_manager.cc:1428] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:22:59.215660 1490 catalog_manager.cc:4841] TS 1ade04976c334268b09b674a3c9197c3 (127.0.98.1:36215): tablet c570a58e86ae4effb343c352570a2f03 (table test-table [id=b34a817e43f543d2b08ccebb1d535040]) successfully deleted
I20260927 03:22:59.219158 2036 ts_tablet_manager.cc:1905] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:22:59.220590 2036 ts_tablet_manager.cc:1918] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:22:59.220654 2036 log.cc:1196] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/c570a58e86ae4effb343c352570a2f03
I20260927 03:22:59.220854 2036 ts_tablet_manager.cc:1939] T c570a58e86ae4effb343c352570a2f03 P a57801eab5e049a4a7a83d318b93d535: Deleting consensus metadata
I20260927 03:22:59.221321 1491 catalog_manager.cc:4841] TS a57801eab5e049a4a7a83d318b93d535 (127.0.98.2:45259): tablet c570a58e86ae4effb343c352570a2f03 (table test-table [id=b34a817e43f543d2b08ccebb1d535040]) successfully deleted
W20260927 03:22:59.252591 1989 tablet.cc:2369] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:22:59.365590 2031 raft_consensus.cc:482] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:22:59.365701 2031 raft_consensus.cc:504] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.365842 2031 leader_election.cc:288] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade04976c334268b09b674a3c9197c3 (127.0.98.1:36215), 6129c2ee946643c685c3697f6301f646 (127.0.98.3:33459)
I20260927 03:22:59.366086 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e25394c0e6049c6a014ebf8125dbc18" candidate_uuid: "a57801eab5e049a4a7a83d318b93d535" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade04976c334268b09b674a3c9197c3" is_pre_election: true
I20260927 03:22:59.366080 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e25394c0e6049c6a014ebf8125dbc18" candidate_uuid: "a57801eab5e049a4a7a83d318b93d535" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6129c2ee946643c685c3697f6301f646" is_pre_election: true
I20260927 03:22:59.366174 1942 raft_consensus.cc:2454] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a57801eab5e049a4a7a83d318b93d535 in term 0.
I20260927 03:22:59.366175 1662 raft_consensus.cc:2454] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a57801eab5e049a4a7a83d318b93d535 in term 0.
I20260927 03:22:59.366335 1733 leader_election.cc:302] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6129c2ee946643c685c3697f6301f646, a57801eab5e049a4a7a83d318b93d535; no voters:
I20260927 03:22:59.366460 2031 raft_consensus.cc:2781] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:22:59.366513 2031 raft_consensus.cc:482] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:22:59.366532 2031 raft_consensus.cc:3037] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:59.367308 2031 raft_consensus.cc:504] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.367439 2031 leader_election.cc:288] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [CANDIDATE]: Term 1 election: Requested vote from peers 1ade04976c334268b09b674a3c9197c3 (127.0.98.1:36215), 6129c2ee946643c685c3697f6301f646 (127.0.98.3:33459)
I20260927 03:22:59.367622 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e25394c0e6049c6a014ebf8125dbc18" candidate_uuid: "a57801eab5e049a4a7a83d318b93d535" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6129c2ee946643c685c3697f6301f646"
I20260927 03:22:59.367635 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e25394c0e6049c6a014ebf8125dbc18" candidate_uuid: "a57801eab5e049a4a7a83d318b93d535" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade04976c334268b09b674a3c9197c3"
I20260927 03:22:59.367698 1942 raft_consensus.cc:3037] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:59.367723 1662 raft_consensus.cc:3037] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:59.368494 1942 raft_consensus.cc:2454] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a57801eab5e049a4a7a83d318b93d535 in term 1.
I20260927 03:22:59.368593 1662 raft_consensus.cc:2454] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a57801eab5e049a4a7a83d318b93d535 in term 1.
I20260927 03:22:59.368722 1733 leader_election.cc:302] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6129c2ee946643c685c3697f6301f646, a57801eab5e049a4a7a83d318b93d535; no voters:
I20260927 03:22:59.368813 2031 raft_consensus.cc:2781] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:22:59.368886 2031 raft_consensus.cc:686] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [term 1 LEADER]: Becoming Leader. State: Replica: a57801eab5e049a4a7a83d318b93d535, State: Running, Role: LEADER
I20260927 03:22:59.368964 2031 consensus_queue.cc:237] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.369557 1502 catalog_manager.cc:5495] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 reported cstate change: term changed from 0 to 1, leader changed from <none> to a57801eab5e049a4a7a83d318b93d535 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a57801eab5e049a4a7a83d318b93d535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } health_report { overall_health: UNKNOWN } } }
Sep 27 03:22:59 dist-test-slave-gw2v krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:22:59.415150 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.407548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40669 (local address 127.0.0.1:47990)
0927 03:22:59.407730 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:59.407741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:59.408273 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:59.408408 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:59.408409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:59.408720 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:22:59.408778 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.408780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.409467 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.409468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.410065 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.410067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.410416 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.410417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.410470 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.411091 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:22:59.411095 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:22:59.412593 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:22:59.414058 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:59.414066 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:59.414068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:59.414271 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:59.414519 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:22:59.414520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:22:59.414520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:22:59.414581 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:22:59.414794 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:22:59.414801 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:22:59.414903 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 03:22:59.414961 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:22:59.415056 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260927 03:22:59.415201 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.407621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47990 (local address 127.0.98.62:40669)
0927 03:22:59.407691 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.407693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.408289 (+ 596us) server_negotiation.cc:407] Connection header received
0927 03:22:59.408326 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.408327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.408338 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.408395 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:59.408870 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.409389 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.410158 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.410349 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.410404 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.412682 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:59.412683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:59.412684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:59.412691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:59.413968 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:59.414357 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:59.414358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:59.414358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:59.414430 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:59.414636 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:59.414636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:59.414637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:59.414745 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:59.414799 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.414989 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.415106 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":38}
I20260927 03:22:59.417296 1502 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47990:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 03:22:59.417503 1502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:22:59.418999 1782 tablet_service.cc:1467] Processing CreateTablet for tablet f8472cca0fa3456685433e96c50ed365 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1fecc6337379405ab896d491fc2375ea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:22:59.419027 1642 tablet_service.cc:1467] Processing CreateTablet for tablet f8472cca0fa3456685433e96c50ed365 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1fecc6337379405ab896d491fc2375ea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:22:59.419065 1922 tablet_service.cc:1467] Processing CreateTablet for tablet f8472cca0fa3456685433e96c50ed365 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1fecc6337379405ab896d491fc2375ea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:22:59.419157 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8472cca0fa3456685433e96c50ed365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:59.419178 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8472cca0fa3456685433e96c50ed365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:59.419178 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8472cca0fa3456685433e96c50ed365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:22:59.420533 2013 tablet_bootstrap.cc:492] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535: Bootstrap starting.
I20260927 03:22:59.420594 2014 tablet_bootstrap.cc:492] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3: Bootstrap starting.
I20260927 03:22:59.420674 2012 tablet_bootstrap.cc:492] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646: Bootstrap starting.
I20260927 03:22:59.420862 2013 tablet_bootstrap.cc:654] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:59.421006 2014 tablet_bootstrap.cc:654] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:59.421027 2012 tablet_bootstrap.cc:654] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646: Neither blocks nor log segments found. Creating new log.
I20260927 03:22:59.421622 2014 tablet_bootstrap.cc:492] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3: No bootstrap required, opened a new log
I20260927 03:22:59.421622 2012 tablet_bootstrap.cc:492] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646: No bootstrap required, opened a new log
I20260927 03:22:59.421672 2014 ts_tablet_manager.cc:1397] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:22:59.421672 2012 ts_tablet_manager.cc:1397] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:22:59.421828 2012 raft_consensus.cc:348] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.421933 2012 raft_consensus.cc:374] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:59.421967 2012 raft_consensus.cc:729] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6129c2ee946643c685c3697f6301f646, State: Initialized, Role: FOLLOWER
I20260927 03:22:59.422032 2012 consensus_queue.cc:260] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.422150 2012 ts_tablet_manager.cc:1428] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:59.422215 2019 tablet_replica.cc:439] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } } }
I20260927 03:22:59.422323 2019 tablet_replica.cc:442] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:22:59.422346 2014 raft_consensus.cc:348] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.422422 2014 raft_consensus.cc:374] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:59.422441 2014 raft_consensus.cc:729] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade04976c334268b09b674a3c9197c3, State: Initialized, Role: FOLLOWER
I20260927 03:22:59.422482 2014 consensus_queue.cc:260] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.422575 2014 ts_tablet_manager.cc:1428] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:22:59.422708 2018 tablet_replica.cc:439] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } } }
I20260927 03:22:59.422799 2018 tablet_replica.cc:442] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:22:59.423316 2013 tablet_bootstrap.cc:492] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535: No bootstrap required, opened a new log
I20260927 03:22:59.423360 2013 ts_tablet_manager.cc:1397] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:22:59.423473 2013 raft_consensus.cc:348] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.423527 2013 raft_consensus.cc:374] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:22:59.423537 2013 raft_consensus.cc:729] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57801eab5e049a4a7a83d318b93d535, State: Initialized, Role: FOLLOWER
I20260927 03:22:59.423570 2013 consensus_queue.cc:260] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.423633 2013 ts_tablet_manager.cc:1428] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:22:59.423674 2031 tablet_replica.cc:439] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } } }
I20260927 03:22:59.423748 2031 tablet_replica.cc:442] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:22:59.449759 2018 raft_consensus.cc:482] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:22:59.449870 2018 raft_consensus.cc:504] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.450217 2018 leader_election.cc:288] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a57801eab5e049a4a7a83d318b93d535 (127.0.98.2:45259), 6129c2ee946643c685c3697f6301f646 (127.0.98.3:33459)
I20260927 03:22:59.453137 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.450467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40729 (local address 127.0.98.3:33459)
0927 03:22:59.450526 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.450528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.450619 (+ 91us) server_negotiation.cc:407] Connection header received
0927 03:22:59.450677 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.450678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.450687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.450728 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:22:59.450910 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.451473 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.452481 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.452831 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.452879 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.452888 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.453033 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.453068 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:22:59.453174 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.450467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50289 (local address 127.0.98.2:45259)
0927 03:22:59.450542 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.450544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.450628 (+ 84us) server_negotiation.cc:407] Connection header received
0927 03:22:59.450692 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.450693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.450699 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.450742 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:22:59.450946 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.451420 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.452518 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.452891 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.452955 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.452965 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.453081 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.453118 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 03:22:59.453229 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.450397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45259 (local address 127.0.98.1:50289)
0927 03:22:59.450565 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:59.450574 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:59.450630 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:59.450753 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:59.450754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:59.450756 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:22:59.450841 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.450844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.451482 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.451483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.452439 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.452441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.452960 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.452961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.453016 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.453020 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:22:59.453070 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:22:59.453104 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":27,"thread_start_us":59,"threads_started":1}
I20260927 03:22:59.453145 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.450397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33459 (local address 127.0.98.1:40729)
0927 03:22:59.450539 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:59.450550 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:59.450616 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:59.450741 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:59.450741 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:59.450743 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:22:59.450841 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.450844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.451549 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.451550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.452411 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.452413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.452895 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.452896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.452968 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.452973 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:22:59.453020 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:22:59.453049 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260927 03:22:59.453555 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8472cca0fa3456685433e96c50ed365" candidate_uuid: "1ade04976c334268b09b674a3c9197c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6129c2ee946643c685c3697f6301f646" is_pre_election: true
I20260927 03:22:59.453526 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8472cca0fa3456685433e96c50ed365" candidate_uuid: "1ade04976c334268b09b674a3c9197c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57801eab5e049a4a7a83d318b93d535" is_pre_election: true
I20260927 03:22:59.453637 1942 raft_consensus.cc:2454] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade04976c334268b09b674a3c9197c3 in term 0.
I20260927 03:22:59.453653 1802 raft_consensus.cc:2454] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade04976c334268b09b674a3c9197c3 in term 0.
I20260927 03:22:59.453825 1595 leader_election.cc:302] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade04976c334268b09b674a3c9197c3, a57801eab5e049a4a7a83d318b93d535; no voters:
I20260927 03:22:59.453964 2018 raft_consensus.cc:2781] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:22:59.454006 2018 raft_consensus.cc:482] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:22:59.454025 2018 raft_consensus.cc:3037] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:59.454682 2018 raft_consensus.cc:504] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.454808 2018 leader_election.cc:288] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [CANDIDATE]: Term 1 election: Requested vote from peers a57801eab5e049a4a7a83d318b93d535 (127.0.98.2:45259), 6129c2ee946643c685c3697f6301f646 (127.0.98.3:33459)
I20260927 03:22:59.454982 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8472cca0fa3456685433e96c50ed365" candidate_uuid: "1ade04976c334268b09b674a3c9197c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57801eab5e049a4a7a83d318b93d535"
I20260927 03:22:59.455065 1802 raft_consensus.cc:3037] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:59.455025 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8472cca0fa3456685433e96c50ed365" candidate_uuid: "1ade04976c334268b09b674a3c9197c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6129c2ee946643c685c3697f6301f646"
I20260927 03:22:59.455120 1942 raft_consensus.cc:3037] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:22:59.455686 1802 raft_consensus.cc:2454] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade04976c334268b09b674a3c9197c3 in term 1.
I20260927 03:22:59.455699 1942 raft_consensus.cc:2454] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade04976c334268b09b674a3c9197c3 in term 1.
I20260927 03:22:59.455868 1595 leader_election.cc:302] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1ade04976c334268b09b674a3c9197c3, 6129c2ee946643c685c3697f6301f646, a57801eab5e049a4a7a83d318b93d535; no voters:
I20260927 03:22:59.455958 2018 raft_consensus.cc:2781] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:22:59.456100 2018 raft_consensus.cc:686] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [term 1 LEADER]: Becoming Leader. State: Replica: 1ade04976c334268b09b674a3c9197c3, State: Running, Role: LEADER
I20260927 03:22:59.456167 2018 consensus_queue.cc:237] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } }
I20260927 03:22:59.456470 2053 tablet_replica.cc:439] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3: TxnStatusTablet state changed. Reason: New leader 1ade04976c334268b09b674a3c9197c3. Latest consensus state: current_term: 1 leader_uuid: "1ade04976c334268b09b674a3c9197c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } } }
I20260927 03:22:59.456544 2053 tablet_replica.cc:442] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:22:59.456696 2055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 03:22:59.456784 2055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 03:22:59.456825 1502 catalog_manager.cc:5495] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade04976c334268b09b674a3c9197c3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1ade04976c334268b09b674a3c9197c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } health_report { overall_health: UNKNOWN } } }
I20260927 03:22:59.470675 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.468128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.0.98.1:36215)
0927 03:22:59.468173 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.468175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.468185 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:22:59.468247 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.468248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.468277 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.468330 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:22:59.468454 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.468924 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.469991 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.470186 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.470268 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.470406 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:22:59.470498 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:22:59.470547 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.470598 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.470616 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:22:59.470731 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.468039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36215 (local address 127.0.0.1:52018)
0927 03:22:59.468106 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:59.468118 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:59.468169 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:59.468344 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:59.468345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:59.468347 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:22:59.468402 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.468405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.468982 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.468983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.469897 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.469900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.470254 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.470255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.470310 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.470327 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:22:59.470550 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:22:59.470555 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:22:59.470597 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 03:22:59.470647 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 03:22:59.471932 1942 raft_consensus.cc:1261] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646 [term 1 FOLLOWER]: Refusing update from remote peer 1ade04976c334268b09b674a3c9197c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:22:59.472011 1802 raft_consensus.cc:1261] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535 [term 1 FOLLOWER]: Refusing update from remote peer 1ade04976c334268b09b674a3c9197c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:22:59.472245 2053 consensus_queue.cc:1035] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:22:59.472389 2018 consensus_queue.cc:1035] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:22:59.473044 2019 tablet_replica.cc:439] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646: TxnStatusTablet state changed. Reason: New leader 1ade04976c334268b09b674a3c9197c3. Latest consensus state: current_term: 1 leader_uuid: "1ade04976c334268b09b674a3c9197c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } } }
I20260927 03:22:59.473142 2019 tablet_replica.cc:442] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:22:59.473402 2031 tablet_replica.cc:439] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535: TxnStatusTablet state changed. Reason: New leader 1ade04976c334268b09b674a3c9197c3. Latest consensus state: current_term: 1 leader_uuid: "1ade04976c334268b09b674a3c9197c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } } }
I20260927 03:22:59.473483 2031 tablet_replica.cc:442] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:22:59.473984 2053 tablet_replica.cc:439] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ade04976c334268b09b674a3c9197c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } } }
I20260927 03:22:59.474068 2053 tablet_replica.cc:442] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:22:59.474185 2019 tablet_replica.cc:439] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ade04976c334268b09b674a3c9197c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } } }
I20260927 03:22:59.474269 2019 tablet_replica.cc:442] T f8472cca0fa3456685433e96c50ed365 P 6129c2ee946643c685c3697f6301f646: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:22:59.474993 2053 tablet_replica.cc:439] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ade04976c334268b09b674a3c9197c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } } }
I20260927 03:22:59.475075 2053 tablet_replica.cc:442] T f8472cca0fa3456685433e96c50ed365 P 1ade04976c334268b09b674a3c9197c3: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:22:59.475489 2025 mvcc.cc:205] Tried to move back new op lower bound from 7333803538315247616 to 7333803538252955648. Current Snapshot: MvccSnapshot[applied={T|T < 7333803538315247616}]
I20260927 03:22:59.475706 2031 tablet_replica.cc:439] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ade04976c334268b09b674a3c9197c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "a57801eab5e049a4a7a83d318b93d535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45259 } } peers { permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 } } }
I20260927 03:22:59.475783 2031 tablet_replica.cc:442] T f8472cca0fa3456685433e96c50ed365 P a57801eab5e049a4a7a83d318b93d535: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:22:59.480054 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.477402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59159 (local address 127.0.98.1:36215)
0927 03:22:59.477443 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.477445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.477518 (+ 73us) server_negotiation.cc:407] Connection header received
0927 03:22:59.477588 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.477588 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.477597 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.477638 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:22:59.477747 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.478243 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.479358 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.479541 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.479594 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.479758 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:22:59.479789 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:22:59.479826 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.479986 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.480003 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:22:59.480121 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.477318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36215 (local address 127.0.98.2:59159)
0927 03:22:59.477449 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:59.477461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:59.477534 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:59.477642 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:59.477643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:59.477645 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:22:59.477694 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.477696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.478307 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.478308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.479272 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.479275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.479608 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.479609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.479669 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.479684 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:22:59.479932 (+ 248us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:22:59.479934 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:22:59.479983 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:22:59.480032 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260927 03:22:59.482184 1662 raft_consensus.cc:1261] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3 [term 1 FOLLOWER]: Refusing update from remote peer a57801eab5e049a4a7a83d318b93d535: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:22:59.482227 1942 raft_consensus.cc:1261] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646 [term 1 FOLLOWER]: Refusing update from remote peer a57801eab5e049a4a7a83d318b93d535: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:22:59.482379 2031 consensus_queue.cc:1035] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade04976c334268b09b674a3c9197c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:22:59.482478 2022 consensus_queue.cc:1035] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6129c2ee946643c685c3697f6301f646" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:22:59.494160 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.491427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47475 (local address 127.0.98.2:45259)
0927 03:22:59.491526 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.491528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.491535 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:22:59.491555 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.491556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.491568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.491620 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:22:59.491747 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.492511 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.493299 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.493597 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.493662 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.493904 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:22:59.493940 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:22:59.493967 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.494066 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.494090 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 03:22:59.494400 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.491224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45259 (local address 127.0.98.1:47475)
0927 03:22:59.491400 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:22:59.491411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:22:59.491458 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:22:59.491622 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:22:59.491623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:22:59.491625 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:22:59.491690 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.491692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.492595 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.492596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.493225 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:22:59.493228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.493737 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.493738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:22:59.493799 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.493812 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:22:59.494006 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:22:59.494007 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:22:59.494272 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 03:22:59.494314 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":102,"threads_started":1}
I20260927 03:22:59.501750 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:22:59.501855 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:22:59.502413 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c6bc9bdc18d4520b425f4fcaf834a31: Sending DeleteTablet for 3 replicas of tablet 5e25394c0e6049c6a014ebf8125dbc18
I20260927 03:22:59.502660 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e25394c0e6049c6a014ebf8125dbc18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:22:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52638
I20260927 03:22:59.502666 1922 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e25394c0e6049c6a014ebf8125dbc18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:22:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56970
I20260927 03:22:59.502744 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e25394c0e6049c6a014ebf8125dbc18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:22:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52016
I20260927 03:22:59.502769 2036 tablet_replica.cc:330] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535: stopping tablet replica
I20260927 03:22:59.502841 2036 raft_consensus.cc:2229] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:22:59.502890 2036 raft_consensus.cc:2258] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:22:59.502960 2037 tablet_replica.cc:330] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646: stopping tablet replica
I20260927 03:22:59.503074 2035 tablet_replica.cc:330] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3: stopping tablet replica
I20260927 03:22:59.503134 2035 raft_consensus.cc:2229] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:22:59.503170 2035 raft_consensus.cc:2258] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:22:59.503233 2037 raft_consensus.cc:2229] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:22:59.503276 2037 raft_consensus.cc:2258] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:22:59.503446 2035 ts_tablet_manager.cc:1905] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:22:59.503602 2036 ts_tablet_manager.cc:1905] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:22:59.504817 2037 ts_tablet_manager.cc:1905] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:22:59.505074 2035 ts_tablet_manager.cc:1918] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:22:59.505098 2036 ts_tablet_manager.cc:1918] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:22:59.505131 2035 log.cc:1196] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-0/wal/wals/5e25394c0e6049c6a014ebf8125dbc18
I20260927 03:22:59.505153 2036 log.cc:1196] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/5e25394c0e6049c6a014ebf8125dbc18
I20260927 03:22:59.505335 2035 ts_tablet_manager.cc:1939] T 5e25394c0e6049c6a014ebf8125dbc18 P 1ade04976c334268b09b674a3c9197c3: Deleting consensus metadata
I20260927 03:22:59.505335 2036 ts_tablet_manager.cc:1939] T 5e25394c0e6049c6a014ebf8125dbc18 P a57801eab5e049a4a7a83d318b93d535: Deleting consensus metadata
I20260927 03:22:59.505635 1491 catalog_manager.cc:4841] TS a57801eab5e049a4a7a83d318b93d535 (127.0.98.2:45259): tablet 5e25394c0e6049c6a014ebf8125dbc18 (table test-table [id=ddbe3de93c724184a04dd359643d5855]) successfully deleted
I20260927 03:22:59.506274 2037 ts_tablet_manager.cc:1918] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:22:59.506289 1490 catalog_manager.cc:4841] TS 1ade04976c334268b09b674a3c9197c3 (127.0.98.1:36215): tablet 5e25394c0e6049c6a014ebf8125dbc18 (table test-table [id=ddbe3de93c724184a04dd359643d5855]) successfully deleted
I20260927 03:22:59.506345 2037 log.cc:1196] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/5e25394c0e6049c6a014ebf8125dbc18
I20260927 03:22:59.506533 2037 ts_tablet_manager.cc:1939] T 5e25394c0e6049c6a014ebf8125dbc18 P 6129c2ee946643c685c3697f6301f646: Deleting consensus metadata
I20260927 03:22:59.507077 1488 catalog_manager.cc:4841] TS 6129c2ee946643c685c3697f6301f646 (127.0.98.3:33459): tablet 5e25394c0e6049c6a014ebf8125dbc18 (table test-table [id=ddbe3de93c724184a04dd359643d5855]) successfully deleted
Sep 27 03:22:59 dist-test-slave-gw2v krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479379, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:22:59.513388 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.503579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52028 (local address 127.0.98.1:36215)
0927 03:22:59.504571 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.504573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.504580 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:22:59.504596 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.504597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.504604 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.504651 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:59.505422 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.506150 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.507313 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.507557 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.507611 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.510759 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:59.510769 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:59.510769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:59.510777 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:59.512370 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:59.512682 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:59.512686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:59.512686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:59.512739 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:59.512903 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:59.512903 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:59.512904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:59.513037 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:59.513181 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.513217 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.513289 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":973}
W20260927 03:22:59.513665 1700 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52028
I20260927 03:22:59.520102 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:22:59.514573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.0.98.62:40669)
0927 03:22:59.514618 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:22:59.514620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:22:59.514669 (+ 49us) server_negotiation.cc:407] Connection header received
0927 03:22:59.514726 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:22:59.514728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:22:59.514737 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:22:59.514780 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:22:59.515228 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.515781 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.516417 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:22:59.516622 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:22:59.516673 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:22:59.517707 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:22:59.517708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:22:59.517709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:22:59.517716 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:22:59.519144 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:59.519438 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:59.519439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:59.519440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:59.519491 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:22:59.519645 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:22:59.519645 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:22:59.519646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:22:59.519770 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:22:59.519854 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:22:59.519946 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 03:22:59.520016 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 03:22:59.520308 1503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47998
I20260927 03:22:59.520824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 1576
I20260927 03:22:59.525722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 1717
I20260927 03:22:59.530405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 1857
I20260927 03:22:59.534493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 1472
2026-09-27T03:22:59Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6376 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:22:59 dist-test-slave-gw2v krb5kdc[2075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:22:59 dist-test-slave-gw2v krb5kdc[2075](info): set up 2 sockets
Sep 27 03:22:59 dist-test-slave-gw2v krb5kdc[2075](info): commencing operation
krb5kdc: starting...
W20260927 03:23:00.494220 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:23:00 dist-test-slave-gw2v krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:23:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:23:00.610261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44377
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:00.690647 2088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:00.690954 2088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:00.690994 2088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:00.692219 2088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:00.692269 2088 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 03:23:00.692286 2088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:00.692302 2088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:00.692317 2088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:00.692332 2088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:00.693867 2088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44377
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:00.694120 2088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:00.694334 2088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:00.699126 2094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:00.699139 2093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:00.699163 2096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:00.699448 2088 server_base.cc:1034] running on GCE node
I20260927 03:23:00.699703 2088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:00.700052 2088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:00.701215 2088 hybrid_clock.cc:648] HybridClock initialized: now 1790479380701188 us; error 42 us; skew 500 ppm
Sep 27 03:23:00 dist-test-slave-gw2v krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:00.703774 2088 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:23:00.704238 2088 webserver.cc:460] Webserver started at http://127.0.98.62:33599/ using document root <none> and password file <none>
I20260927 03:23:00.704432 2088 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:00.704473 2088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:00.704563 2088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:00.705731 2088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "594472a6516041b79e2f17a0cbf8ee9b"
format_stamp: "Formatted at 2026-09-27 03:23:00 on dist-test-slave-gw2v"
server_key: "250560a4d5755fc2fb743af08bca3f66"
server_key_iv: "d447d7d15d1efe0872983805d06db23c"
server_key_version: "encryptionkey@0"
I20260927 03:23:00.706044 2088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "594472a6516041b79e2f17a0cbf8ee9b"
format_stamp: "Formatted at 2026-09-27 03:23:00 on dist-test-slave-gw2v"
server_key: "250560a4d5755fc2fb743af08bca3f66"
server_key_iv: "d447d7d15d1efe0872983805d06db23c"
server_key_version: "encryptionkey@0"
I20260927 03:23:00.707434 2088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:23:00.708181 2103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:00.708374 2088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:00.708463 2088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "594472a6516041b79e2f17a0cbf8ee9b"
format_stamp: "Formatted at 2026-09-27 03:23:00 on dist-test-slave-gw2v"
server_key: "250560a4d5755fc2fb743af08bca3f66"
server_key_iv: "d447d7d15d1efe0872983805d06db23c"
server_key_version: "encryptionkey@0"
I20260927 03:23:00.708528 2088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:00.719913 2088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:00.722045 2088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:00.722153 2088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:00.726351 2088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44377
I20260927 03:23:00.726462 2165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44377 every 8 connection(s)
I20260927 03:23:00.726708 2088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:00.727404 2166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:00.729583 2166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b: Bootstrap starting.
I20260927 03:23:00.730113 2166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:00.730329 2166 log.cc:824] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:00.731066 2166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b: No bootstrap required, opened a new log
I20260927 03:23:00.733035 2166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594472a6516041b79e2f17a0cbf8ee9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44377 } }
I20260927 03:23:00.733170 2166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:00.733191 2166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594472a6516041b79e2f17a0cbf8ee9b, State: Initialized, Role: FOLLOWER
I20260927 03:23:00.733330 2166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594472a6516041b79e2f17a0cbf8ee9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44377 } }
I20260927 03:23:00.733397 2166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:00.733418 2166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:00.733461 2166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:00.733970 2166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594472a6516041b79e2f17a0cbf8ee9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44377 } }
I20260927 03:23:00.734054 2166 leader_election.cc:302] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 594472a6516041b79e2f17a0cbf8ee9b; no voters:
I20260927 03:23:00.734198 2166 leader_election.cc:288] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:00.734256 2169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:00.734419 2169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [term 1 LEADER]: Becoming Leader. State: Replica: 594472a6516041b79e2f17a0cbf8ee9b, State: Running, Role: LEADER
I20260927 03:23:00.734438 2166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:00.734551 2169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594472a6516041b79e2f17a0cbf8ee9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44377 } }
I20260927 03:23:00.734921 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 594472a6516041b79e2f17a0cbf8ee9b. Latest consensus state: current_term: 1 leader_uuid: "594472a6516041b79e2f17a0cbf8ee9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594472a6516041b79e2f17a0cbf8ee9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44377 } } }
I20260927 03:23:00.734997 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:00.735038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 2088
I20260927 03:23:00.735114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:00.735142 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "594472a6516041b79e2f17a0cbf8ee9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594472a6516041b79e2f17a0cbf8ee9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44377 } } }
I20260927 03:23:00.735184 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:00.735276 392 external_mini_cluster.cc:1428] Setting key 0f2f4a8eff5f75e8d15e10daa1e0154c
I20260927 03:23:00.735816 2178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:00.736425 2178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:00.737989 2178 catalog_manager.cc:1347] Generated new cluster ID: bff5feeafb53426c95b1944c54614b9b
I20260927 03:23:00.738041 2178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:23:00 dist-test-slave-gw2v krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:00.746377 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:00.735724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.0.98.62:44377)
0927 03:23:00.736057 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:00.736066 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:00.736095 (+ 29us) server_negotiation.cc:407] Connection header received
0927 03:23:00.736216 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:00.736224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:00.736281 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:00.736429 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:00.737235 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:00.738413 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:00.739498 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:00.740097 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:00.740191 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:00.742895 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:00.742899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:00.742900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:00.742913 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:00.744805 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:00.745157 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:00.745161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:00.745162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:00.745289 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:00.745462 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:00.745463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:00.745464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:00.745790 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:00.745936 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:00.746015 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 03:23:00.746153 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":81,"threads_started":1}
I20260927 03:23:00.760489 2178 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:00.761015 2178 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:00.766047 2178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b: Generated new TSK 0
I20260927 03:23:00.766215 2178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:23:00.794503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44377
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:00.875505 2192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:00.875720 2192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:00.875746 2192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:00.876842 2192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:00.876875 2192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:00.876893 2192 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 03:23:00.876904 2192 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:23:00.876919 2192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:23:00.878316 2192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:00.878501 2192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:00.878692 2192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:00.883595 2200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:00.883613 2197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:00.883589 2198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:00.883975 2192 server_base.cc:1034] running on GCE node
I20260927 03:23:00.884179 2192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:00.884430 2192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:00.885584 2192 hybrid_clock.cc:648] HybridClock initialized: now 1790479380885572 us; error 28 us; skew 500 ppm
Sep 27 03:23:00 dist-test-slave-gw2v krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:00.888286 2192 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:23:00.888780 2192 webserver.cc:460] Webserver started at http://127.0.98.1:35923/ using document root <none> and password file <none>
I20260927 03:23:00.888991 2192 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:00.889051 2192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:00.889168 2192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:00.890271 2192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "6cccd6746a5548a3af3923170e8e0938"
format_stamp: "Formatted at 2026-09-27 03:23:00 on dist-test-slave-gw2v"
server_key: "b7d68495f4f4a96006785750cd455305"
server_key_iv: "25e662de7a053a31ce356f55c3e9972e"
server_key_version: "encryptionkey@0"
I20260927 03:23:00.890612 2192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6cccd6746a5548a3af3923170e8e0938"
format_stamp: "Formatted at 2026-09-27 03:23:00 on dist-test-slave-gw2v"
server_key: "b7d68495f4f4a96006785750cd455305"
server_key_iv: "25e662de7a053a31ce356f55c3e9972e"
server_key_version: "encryptionkey@0"
I20260927 03:23:00.891983 2192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:00.892735 2207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:00.892894 2192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:00.892976 2192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "6cccd6746a5548a3af3923170e8e0938"
format_stamp: "Formatted at 2026-09-27 03:23:00 on dist-test-slave-gw2v"
server_key: "b7d68495f4f4a96006785750cd455305"
server_key_iv: "25e662de7a053a31ce356f55c3e9972e"
server_key_version: "encryptionkey@0"
I20260927 03:23:00.893042 2192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:00.907814 2192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:00.910126 2192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:00.910269 2192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:00.910897 2192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:00.910948 2192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:00.910979 2192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:00.911027 2192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:23:00 dist-test-slave-gw2v krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:00.917758 2192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43593
I20260927 03:23:00.917830 2322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43593 every 8 connection(s)
I20260927 03:23:00.918174 2192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:23:00.919494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 2192
I20260927 03:23:00.919595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:23:00.919795 392 external_mini_cluster.cc:1428] Setting key 9dfcaebfdede834a2c527d7ae76f792f
I20260927 03:23:00.920228 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:00.911235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34135 (local address 127.0.98.62:44377)
0927 03:23:00.911280 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:00.911282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:00.911579 (+ 297us) server_negotiation.cc:407] Connection header received
0927 03:23:00.912190 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:00.912192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:00.912213 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:00.912255 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:00.912922 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:00.913439 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:00.914457 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:00.914662 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:00.914718 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:00.917175 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:00.917177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:00.917177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:00.917187 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:00.918677 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:00.919120 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:00.919122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:00.919122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:00.919183 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:00.919392 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:00.919393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:00.919393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:00.919515 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:00.919573 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:00.920034 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 03:23:00.920126 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:23:00.920393 2216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:00.911266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44377 (local address 127.0.98.1:34135)
0927 03:23:00.911452 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:00.911492 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:00.912084 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:00.912359 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:00.912363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:00.912623 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:00.912824 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:00.912828 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:00.913521 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:00.913522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:00.914365 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:00.914368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:00.914735 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:00.914736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:00.914941 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:00.915470 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:00.915480 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:00.917078 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:00.918805 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:00.918812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:00.918832 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:00.919053 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:00.919236 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:00.919237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:00.919237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:00.919340 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:00.919601 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:00.919608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:00.919883 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 03:23:00.920009 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 03:23:00.920175 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260927 03:23:00.920863 2323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44377
I20260927 03:23:00.920948 2323 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:00.921120 2323 heartbeater.cc:511] Master 127.0.98.62:44377 requested a full tablet report, sending...
I20260927 03:23:00.921649 2119 ts_manager.cc:194] Registered new tserver with Master: 6cccd6746a5548a3af3923170e8e0938 (127.0.98.1:43593)
I20260927 03:23:00.922683 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34135
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 03:23:00.929844 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:00.923850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33227 (local address 127.0.98.62:44377)
0927 03:23:00.923893 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:00.923895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:00.923955 (+ 60us) server_negotiation.cc:407] Connection header received
0927 03:23:00.924026 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:00.924027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:00.924034 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:00.924088 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:00.924459 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:00.925007 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:00.926039 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:00.926241 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:00.926297 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:00.927345 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:00.927346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:00.927346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:00.927353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:00.928747 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:00.929166 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:00.929167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:00.929167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:00.929227 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:00.929383 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:00.929384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:00.929384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:00.929492 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:00.929537 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:00.929694 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 03:23:00.929769 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:23:00.929868 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:00.923756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44377 (local address 127.0.98.1:33227)
0927 03:23:00.923896 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:00.923908 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:00.923970 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:00.924105 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:00.924106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:00.924309 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:00.924377 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:00.924379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:00.925103 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:00.925104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:00.925939 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:00.925943 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:00.926315 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:00.926316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:00.926390 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:00.926870 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:00.926876 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:00.927242 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:00.928878 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:00.928879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:00.928880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:00.929065 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:00.929283 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:00.929284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:00.929284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:00.929338 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:00.929541 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:00.929542 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:00.929581 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 03:23:00.929675 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 03:23:00.929753 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:23:00.946436 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44377
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:01.029681 2332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:01.029893 2332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:01.029920 2332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:01.031244 2332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:01.031291 2332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:01.031311 2332 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 03:23:01.031320 2332 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:23:01.031343 2332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:23:01.032860 2332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:01.033088 2332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:01.033305 2332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:01.038445 2337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:01.038442 2338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:01.038458 2340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:01.038677 2332 server_base.cc:1034] running on GCE node
I20260927 03:23:01.038939 2332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:01.039188 2332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:01.040339 2332 hybrid_clock.cc:648] HybridClock initialized: now 1790479381040316 us; error 38 us; skew 500 ppm
Sep 27 03:23:01 dist-test-slave-gw2v krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:01.042900 2332 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:23:01.043357 2332 webserver.cc:460] Webserver started at http://127.0.98.2:40089/ using document root <none> and password file <none>
I20260927 03:23:01.043552 2332 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:01.043591 2332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:01.043686 2332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:01.044701 2332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "7b11085737f74876ab2499caefa8e665"
format_stamp: "Formatted at 2026-09-27 03:23:01 on dist-test-slave-gw2v"
server_key: "f241841fe5de6252a3274cb1e1fc3937"
server_key_iv: "28a5b8d00e102320589762c023a8bf39"
server_key_version: "encryptionkey@0"
I20260927 03:23:01.044996 2332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7b11085737f74876ab2499caefa8e665"
format_stamp: "Formatted at 2026-09-27 03:23:01 on dist-test-slave-gw2v"
server_key: "f241841fe5de6252a3274cb1e1fc3937"
server_key_iv: "28a5b8d00e102320589762c023a8bf39"
server_key_version: "encryptionkey@0"
I20260927 03:23:01.046243 2332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:23:01.046934 2347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:01.047086 2332 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:01.047168 2332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "7b11085737f74876ab2499caefa8e665"
format_stamp: "Formatted at 2026-09-27 03:23:01 on dist-test-slave-gw2v"
server_key: "f241841fe5de6252a3274cb1e1fc3937"
server_key_iv: "28a5b8d00e102320589762c023a8bf39"
server_key_version: "encryptionkey@0"
I20260927 03:23:01.047233 2332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:01.060443 2332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:01.062614 2332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:01.062768 2332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:01.063372 2332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:01.063423 2332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:01.063452 2332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:01.063486 2332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:01.070056 2332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36457
I20260927 03:23:01.070137 2462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36457 every 8 connection(s)
Sep 27 03:23:01 dist-test-slave-gw2v krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:01.070456 2332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:23:01.071728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 2332
I20260927 03:23:01.071822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:23:01.072002 392 external_mini_cluster.cc:1428] Setting key d86bae35cff44878890d669bcbd6131d
I20260927 03:23:01.073774 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.063744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59941 (local address 127.0.98.62:44377)
0927 03:23:01.063806 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.063809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.064032 (+ 223us) server_negotiation.cc:407] Connection header received
0927 03:23:01.064532 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.064533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.064546 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.064626 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:01.065488 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.066238 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.067834 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.068026 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.068075 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.070886 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:01.070888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:01.070889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:01.070897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:01.072401 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:01.072874 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:01.072876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:01.072876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:01.072940 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:01.073155 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:01.073156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:01.073156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:01.073291 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:01.073350 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.073585 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.073683 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:23:01.073877 2356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.063739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44377 (local address 127.0.98.2:59941)
0927 03:23:01.063910 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.063951 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.064441 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.064710 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.064714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.065082 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:01.065381 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.065387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.066362 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.066363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.067721 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.067725 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.068107 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.068108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.068267 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.068938 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:01.068948 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:01.070748 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:01.072475 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:01.072483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:01.072486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:01.072800 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:01.073000 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:01.073000 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:01.073001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:01.073105 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:01.073343 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:01.073347 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:01.073469 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.073549 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.073681 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260927 03:23:01.074352 2463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44377
I20260927 03:23:01.074443 2463 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:01.074636 2463 heartbeater.cc:511] Master 127.0.98.62:44377 requested a full tablet report, sending...
I20260927 03:23:01.075038 2119 ts_manager.cc:194] Registered new tserver with Master: 7b11085737f74876ab2499caefa8e665 (127.0.98.2:36457)
I20260927 03:23:01.075413 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59941
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 03:23:01.084936 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.079136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44377 (local address 127.0.98.2:45781)
0927 03:23:01.079321 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.079333 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.079402 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.079532 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.079532 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.079810 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:01.079889 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.079891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.080639 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.080640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.081158 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.081160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.081494 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.081495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.081570 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.081988 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:01.081993 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:01.082287 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:01.083920 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:01.083921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:01.083921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:01.084085 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:01.084276 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:01.084277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:01.084277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:01.084344 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:01.084602 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:01.084603 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:01.084649 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.084713 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.084820 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260927 03:23:01.084995 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.079214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45781 (local address 127.0.98.62:44377)
0927 03:23:01.079268 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.079269 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.079393 (+ 124us) server_negotiation.cc:407] Connection header received
0927 03:23:01.079468 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.079469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.079480 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.079527 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:01.079969 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.080538 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.081239 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.081428 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.081484 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.082373 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:01.082375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:01.082375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:01.082383 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:01.083836 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:01.084148 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:01.084149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:01.084150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:01.084213 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:01.084402 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:01.084403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:01.084404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:01.084549 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:01.084601 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.084853 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.084923 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:23:01.099167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44377
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:01.178565 2472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:01.178777 2472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:01.178802 2472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:01.179931 2472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:01.179972 2472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:01.179991 2472 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 03:23:01.180002 2472 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:23:01.180017 2472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:23:01.181370 2472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:01.181551 2472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:01.181721 2472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:01.186540 2478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:01.186537 2477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:01.186697 2480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:01.186873 2472 server_base.cc:1034] running on GCE node
I20260927 03:23:01.187075 2472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:01.187304 2472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:01.188443 2472 hybrid_clock.cc:648] HybridClock initialized: now 1790479381188432 us; error 30 us; skew 500 ppm
Sep 27 03:23:01 dist-test-slave-gw2v krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:01.191049 2472 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:23:01.191514 2472 webserver.cc:460] Webserver started at http://127.0.98.3:34991/ using document root <none> and password file <none>
I20260927 03:23:01.191725 2472 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:01.191768 2472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:01.191881 2472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:01.192950 2472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "39b23280a2a94ff6b7c3be2b8436b2c9"
format_stamp: "Formatted at 2026-09-27 03:23:01 on dist-test-slave-gw2v"
server_key: "4870ff8a0d07bfa2bd4d08c9cbabab79"
server_key_iv: "660f9e01fd46d9e6f80ae90cbc5c972c"
server_key_version: "encryptionkey@0"
I20260927 03:23:01.193277 2472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "39b23280a2a94ff6b7c3be2b8436b2c9"
format_stamp: "Formatted at 2026-09-27 03:23:01 on dist-test-slave-gw2v"
server_key: "4870ff8a0d07bfa2bd4d08c9cbabab79"
server_key_iv: "660f9e01fd46d9e6f80ae90cbc5c972c"
server_key_version: "encryptionkey@0"
I20260927 03:23:01.194543 2472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:23:01.195345 2487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:01.195492 2472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:01.195568 2472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "39b23280a2a94ff6b7c3be2b8436b2c9"
format_stamp: "Formatted at 2026-09-27 03:23:01 on dist-test-slave-gw2v"
server_key: "4870ff8a0d07bfa2bd4d08c9cbabab79"
server_key_iv: "660f9e01fd46d9e6f80ae90cbc5c972c"
server_key_version: "encryptionkey@0"
I20260927 03:23:01.195633 2472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:01.203119 2472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:01.205137 2472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:01.205264 2472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:01.205768 2472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:01.205806 2472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:01.205827 2472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:01.205838 2472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:01.212646 2472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44631
I20260927 03:23:01.212715 2602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44631 every 8 connection(s)
Sep 27 03:23:01 dist-test-slave-gw2v krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:01.213047 2472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:23:01.213814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 2472
I20260927 03:23:01.213910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:23:01.214119 392 external_mini_cluster.cc:1428] Setting key 625ad5a0272d9588976722e3e1818153
I20260927 03:23:01.215905 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.206146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46429 (local address 127.0.98.62:44377)
0927 03:23:01.206208 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.206210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.206431 (+ 221us) server_negotiation.cc:407] Connection header received
0927 03:23:01.207125 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.207126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.207136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.207194 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:01.208057 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.208667 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.210190 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.210404 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.210482 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.213384 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:01.213386 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:01.213386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:01.213394 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:01.214812 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:01.215214 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:01.215214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:01.215215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:01.215266 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:01.215428 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:01.215429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:01.215429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:01.215535 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:01.215577 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.215779 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.215842 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:23:01.216069 2496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.206110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44377 (local address 127.0.98.3:46429)
0927 03:23:01.206302 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.206351 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.207026 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.207279 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.207284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.207632 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:01.207942 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.207949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.208755 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.208757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.210094 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.210097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.210525 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.210526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.210648 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.211727 (+ 1079us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:01.211736 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:01.213261 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:01.214912 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:01.214919 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:01.214922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:01.215157 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:01.215305 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:01.215305 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:01.215306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:01.215391 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:01.215569 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:01.215571 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:01.215690 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.215763 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.215882 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":99,"threads_started":1}
I20260927 03:23:01.216465 2603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44377
I20260927 03:23:01.216544 2603 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:01.216723 2603 heartbeater.cc:511] Master 127.0.98.62:44377 requested a full tablet report, sending...
I20260927 03:23:01.217070 2120 ts_manager.cc:194] Registered new tserver with Master: 39b23280a2a94ff6b7c3be2b8436b2c9 (127.0.98.3:44631)
I20260927 03:23:01.217450 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46429
I20260927 03:23:01.222327 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.217470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35453 (local address 127.0.98.62:44377)
0927 03:23:01.217510 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.217511 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.217631 (+ 120us) server_negotiation.cc:407] Connection header received
0927 03:23:01.217681 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.217682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.217692 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.217741 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:01.218094 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.218543 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.219308 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.219483 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.219531 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.220329 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:01.220331 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:01.220331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:01.220336 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:01.221499 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:01.221802 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:01.221802 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:01.221803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:01.221850 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:01.221978 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:01.221978 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:01.221979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:01.222065 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:01.222107 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.222200 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.222267 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:23:01.222340 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.217416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44377 (local address 127.0.98.3:35453)
0927 03:23:01.217560 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.217569 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.217629 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.217747 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.217748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.217948 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:01.218018 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.218021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.218689 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.218690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.219246 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.219248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.219559 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.219560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.219636 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.219997 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:01.220001 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:01.220259 (+ 258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:01.221582 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:01.221583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:01.221583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:01.221737 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:01.221895 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:01.221896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:01.221896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:01.221940 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:01.222103 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:01.222104 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:01.222136 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.222179 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.222263 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260927 03:23:01.225641 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:23:01 dist-test-slave-gw2v krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:23:01 dist-test-slave-gw2v krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:01.247889 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.241322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57686 (local address 127.0.98.62:44377)
0927 03:23:01.241375 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.241377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.241386 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:23:01.241412 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.241413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.241421 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.241468 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:01.242211 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.242695 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.243446 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.243628 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.243682 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.245767 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:01.245769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:01.245769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:01.245775 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:01.246923 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:01.247239 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:01.247240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:01.247240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:01.247287 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:01.247474 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:01.247474 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:01.247475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:01.247567 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:01.247635 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.247766 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.247822 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:23:01.249395 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:23:01.251063 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:23:01.257161 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.252977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43593 (local address 127.0.0.1:44080)
0927 03:23:01.253526 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.253531 (+ 5us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.253566 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.253755 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.253756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.253758 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:01.253823 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.253825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.254684 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.254685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.255944 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.255947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.256873 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.256874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.256936 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.256941 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.256986 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.257061 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260927 03:23:01.257153 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.253110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.0.98.3:44631)
0927 03:23:01.253262 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.253268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.253310 (+ 42us) server_negotiation.cc:407] Connection header received
0927 03:23:01.253430 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.253433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.253448 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.253526 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:01.253930 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.255077 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.256162 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.256756 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.256823 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.256882 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.257010 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.257055 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260927 03:23:01.257390 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.252969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44631 (local address 127.0.0.1:36332)
0927 03:23:01.253215 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.253255 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.253331 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.253540 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.253543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.253545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:01.253843 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.253845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.255159 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.255160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.256083 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.256085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.256824 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.256825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.256911 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.256926 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.256989 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.257062 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":66,"thread_start_us":69,"threads_started":1}
I20260927 03:23:01.257428 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.253288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.0.98.2:36457)
0927 03:23:01.253631 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.253635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.253644 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:23:01.253681 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.253683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.253697 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.253764 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:01.253928 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.255046 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.256310 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.256995 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.257053 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.257102 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.257278 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.257331 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":73,"threads_started":1}
I20260927 03:23:01.257474 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.252973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36457 (local address 127.0.0.1:49092)
0927 03:23:01.253439 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.253446 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.253479 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.253826 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.253827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.253829 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:01.253860 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.253861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.255154 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.255155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.256236 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.256238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.257104 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.257105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.257157 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.257160 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.257341 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.257379 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":27,"thread_start_us":58,"threads_started":1}
I20260927 03:23:01.257683 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.253233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.0.98.1:43593)
0927 03:23:01.253419 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.253425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.253578 (+ 153us) server_negotiation.cc:407] Connection header received
0927 03:23:01.253618 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.253620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.253634 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.253750 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:01.253899 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.254604 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.256028 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.256716 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.256782 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.256840 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.257553 (+ 713us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.257593 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
I20260927 03:23:01.258080 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 0d3fe8b88a664af2ae1594fe0f301b2a (DEFAULT_TABLE table=test-table [id=13cffc94590e4c81af74d22ad2f537bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:01.258144 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 0d3fe8b88a664af2ae1594fe0f301b2a (DEFAULT_TABLE table=test-table [id=13cffc94590e4c81af74d22ad2f537bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:01.258400 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d3fe8b88a664af2ae1594fe0f301b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:01.258455 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d3fe8b88a664af2ae1594fe0f301b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:01.260833 2623 tablet_bootstrap.cc:492] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9: Bootstrap starting.
I20260927 03:23:01.261196 2624 tablet_bootstrap.cc:492] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665: Bootstrap starting.
I20260927 03:23:01.261660 2623 tablet_bootstrap.cc:654] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:01.261601 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 0d3fe8b88a664af2ae1594fe0f301b2a (DEFAULT_TABLE table=test-table [id=13cffc94590e4c81af74d22ad2f537bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:01.261757 2624 tablet_bootstrap.cc:654] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:01.261929 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d3fe8b88a664af2ae1594fe0f301b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:01.261981 2623 log.cc:824] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:01.262027 2624 log.cc:824] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:01.262631 2623 tablet_bootstrap.cc:492] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9: No bootstrap required, opened a new log
I20260927 03:23:01.262701 2623 ts_tablet_manager.cc:1397] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:23:01.263152 2624 tablet_bootstrap.cc:492] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665: No bootstrap required, opened a new log
I20260927 03:23:01.263232 2624 ts_tablet_manager.cc:1397] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:23:01.264588 2623 raft_consensus.cc:348] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } }
I20260927 03:23:01.264765 2623 raft_consensus.cc:374] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:01.264806 2623 raft_consensus.cc:729] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b23280a2a94ff6b7c3be2b8436b2c9, State: Initialized, Role: FOLLOWER
I20260927 03:23:01.264729 2627 tablet_bootstrap.cc:492] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938: Bootstrap starting.
I20260927 03:23:01.264914 2623 consensus_queue.cc:260] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } }
I20260927 03:23:01.265165 2623 ts_tablet_manager.cc:1428] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:23:01.265082 2624 raft_consensus.cc:348] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } }
I20260927 03:23:01.265239 2624 raft_consensus.cc:374] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:01.265239 2603 heartbeater.cc:503] Master 127.0.98.62:44377 was elected leader, sending a full tablet report...
I20260927 03:23:01.265281 2624 raft_consensus.cc:729] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b11085737f74876ab2499caefa8e665, State: Initialized, Role: FOLLOWER
I20260927 03:23:01.265412 2624 consensus_queue.cc:260] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } }
I20260927 03:23:01.265667 2624 ts_tablet_manager.cc:1428] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:23:01.265789 2463 heartbeater.cc:503] Master 127.0.98.62:44377 was elected leader, sending a full tablet report...
I20260927 03:23:01.266026 2627 tablet_bootstrap.cc:654] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:01.266350 2627 log.cc:824] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:01.267009 2627 tablet_bootstrap.cc:492] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938: No bootstrap required, opened a new log
I20260927 03:23:01.267059 2627 ts_tablet_manager.cc:1397] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:23:01.268384 2627 raft_consensus.cc:348] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } }
I20260927 03:23:01.268544 2627 raft_consensus.cc:374] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:01.268575 2627 raft_consensus.cc:729] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cccd6746a5548a3af3923170e8e0938, State: Initialized, Role: FOLLOWER
I20260927 03:23:01.268658 2627 consensus_queue.cc:260] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } }
I20260927 03:23:01.268862 2627 ts_tablet_manager.cc:1428] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:23:01.268930 2323 heartbeater.cc:503] Master 127.0.98.62:44377 was elected leader, sending a full tablet report...
W20260927 03:23:01.321091 2464 tablet.cc:2369] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:23:01.389187 2629 raft_consensus.cc:482] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:01.389326 2629 raft_consensus.cc:504] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } }
I20260927 03:23:01.389667 2629 leader_election.cc:288] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39b23280a2a94ff6b7c3be2b8436b2c9 (127.0.98.3:44631), 6cccd6746a5548a3af3923170e8e0938 (127.0.98.1:43593)
I20260927 03:23:01.392554 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.389971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53009 (local address 127.0.98.3:44631)
0927 03:23:01.390082 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.390084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.390119 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:23:01.390198 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.390199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.390209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.390255 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:01.390396 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.390942 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.391904 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.392255 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.392314 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.392325 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.392452 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.392486 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 03:23:01.392597 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.389990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43095 (local address 127.0.98.1:43593)
0927 03:23:01.390058 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.390060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.390079 (+ 19us) server_negotiation.cc:407] Connection header received
0927 03:23:01.390138 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.390139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.390152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.390207 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:01.390354 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.391002 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.391910 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.392269 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.392338 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.392348 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.392478 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.392516 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:23:01.392608 2356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.389859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43593 (local address 127.0.98.2:43095)
0927 03:23:01.389949 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.389963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.390035 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.390219 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.390220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.390222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:01.390294 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.390295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.391094 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.391095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.391824 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.391826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.392345 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.392346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.392414 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.392417 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.392467 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.392509 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":5}
I20260927 03:23:01.392623 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.389859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44631 (local address 127.0.98.2:53009)
0927 03:23:01.390062 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.390073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.390129 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.390269 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.390269 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.390271 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:01.390336 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.390338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.391023 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.391024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.391828 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.391830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.392323 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.392323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.392390 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.392394 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.392439 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.392480 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":66,"thread_start_us":94,"threads_started":1}
I20260927 03:23:01.392992 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3fe8b88a664af2ae1594fe0f301b2a" candidate_uuid: "7b11085737f74876ab2499caefa8e665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cccd6746a5548a3af3923170e8e0938" is_pre_election: true
I20260927 03:23:01.393026 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3fe8b88a664af2ae1594fe0f301b2a" candidate_uuid: "7b11085737f74876ab2499caefa8e665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" is_pre_election: true
I20260927 03:23:01.393160 2277 raft_consensus.cc:2454] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b11085737f74876ab2499caefa8e665 in term 0.
I20260927 03:23:01.393175 2550 raft_consensus.cc:2454] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b11085737f74876ab2499caefa8e665 in term 0.
I20260927 03:23:01.393389 2349 leader_election.cc:302] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b23280a2a94ff6b7c3be2b8436b2c9, 7b11085737f74876ab2499caefa8e665; no voters:
I20260927 03:23:01.393520 2629 raft_consensus.cc:2781] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:23:01.393566 2629 raft_consensus.cc:482] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:01.393579 2629 raft_consensus.cc:3037] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:01.394428 2629 raft_consensus.cc:504] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } }
I20260927 03:23:01.394553 2629 leader_election.cc:288] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [CANDIDATE]: Term 1 election: Requested vote from peers 39b23280a2a94ff6b7c3be2b8436b2c9 (127.0.98.3:44631), 6cccd6746a5548a3af3923170e8e0938 (127.0.98.1:43593)
I20260927 03:23:01.394702 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3fe8b88a664af2ae1594fe0f301b2a" candidate_uuid: "7b11085737f74876ab2499caefa8e665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9"
I20260927 03:23:01.394778 2550 raft_consensus.cc:3037] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:01.394805 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3fe8b88a664af2ae1594fe0f301b2a" candidate_uuid: "7b11085737f74876ab2499caefa8e665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cccd6746a5548a3af3923170e8e0938"
I20260927 03:23:01.394915 2277 raft_consensus.cc:3037] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:01.395505 2550 raft_consensus.cc:2454] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b11085737f74876ab2499caefa8e665 in term 1.
I20260927 03:23:01.395666 2349 leader_election.cc:302] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b23280a2a94ff6b7c3be2b8436b2c9, 7b11085737f74876ab2499caefa8e665; no voters:
I20260927 03:23:01.395728 2277 raft_consensus.cc:2454] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b11085737f74876ab2499caefa8e665 in term 1.
I20260927 03:23:01.395743 2629 raft_consensus.cc:2781] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:01.395892 2629 raft_consensus.cc:686] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [term 1 LEADER]: Becoming Leader. State: Replica: 7b11085737f74876ab2499caefa8e665, State: Running, Role: LEADER
I20260927 03:23:01.395985 2629 consensus_queue.cc:237] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } }
I20260927 03:23:01.396862 2119 catalog_manager.cc:5495] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b11085737f74876ab2499caefa8e665 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7b11085737f74876ab2499caefa8e665" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } health_report { overall_health: UNKNOWN } } }
W20260927 03:23:01.418962 2324 tablet.cc:2369] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 03:23:01 dist-test-slave-gw2v krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:01.438055 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.427504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44377 (local address 127.0.0.1:57698)
0927 03:23:01.427678 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.427687 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.427757 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.427881 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.427882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.428204 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:01.428253 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.428255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.428914 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.428915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.429452 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.429454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.429759 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.429759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.429803 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.433869 (+ 4066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:01.433874 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:01.435417 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:01.437042 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:01.437051 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:01.437053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:01.437254 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:01.437428 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:01.437428 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:01.437429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:01.437484 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:01.437702 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:01.437710 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:01.437756 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.437813 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.437936 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
I20260927 03:23:01.438103 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.427558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.0.98.62:44377)
0927 03:23:01.427616 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.427619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.427748 (+ 129us) server_negotiation.cc:407] Connection header received
0927 03:23:01.427815 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.427816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.427827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.427872 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:01.428327 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.428841 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.429523 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.429701 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.429755 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.435509 (+ 5754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:01.435510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:01.435511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:01.435518 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:01.436964 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:01.437327 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:01.437328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:01.437328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:01.437379 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:01.437529 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:01.437530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:01.437530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:01.437651 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:01.437705 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.437844 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.437994 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":32}
I20260927 03:23:01.440075 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57698:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 03:23:01.440286 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:23:01.441649 2257 tablet_service.cc:1467] Processing CreateTablet for tablet faf53425b9a94f7596cd0616e145fb6b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=902831eeddbe43258c5e97d942b9d3e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:23:01.441744 2397 tablet_service.cc:1467] Processing CreateTablet for tablet faf53425b9a94f7596cd0616e145fb6b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=902831eeddbe43258c5e97d942b9d3e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:23:01.441817 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf53425b9a94f7596cd0616e145fb6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:01.441854 2537 tablet_service.cc:1467] Processing CreateTablet for tablet faf53425b9a94f7596cd0616e145fb6b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=902831eeddbe43258c5e97d942b9d3e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:23:01.441901 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf53425b9a94f7596cd0616e145fb6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:01.441967 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf53425b9a94f7596cd0616e145fb6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:01.442938 2627 tablet_bootstrap.cc:492] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938: Bootstrap starting.
I20260927 03:23:01.443279 2627 tablet_bootstrap.cc:654] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:01.443377 2624 tablet_bootstrap.cc:492] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665: Bootstrap starting.
I20260927 03:23:01.443768 2624 tablet_bootstrap.cc:654] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:01.444080 2623 tablet_bootstrap.cc:492] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9: Bootstrap starting.
I20260927 03:23:01.444198 2627 tablet_bootstrap.cc:492] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938: No bootstrap required, opened a new log
I20260927 03:23:01.444283 2627 ts_tablet_manager.cc:1397] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:01.444351 2624 tablet_bootstrap.cc:492] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665: No bootstrap required, opened a new log
I20260927 03:23:01.444404 2624 ts_tablet_manager.cc:1397] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:01.444446 2627 raft_consensus.cc:348] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.444527 2627 raft_consensus.cc:374] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:01.444555 2627 raft_consensus.cc:729] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cccd6746a5548a3af3923170e8e0938, State: Initialized, Role: FOLLOWER
I20260927 03:23:01.444531 2624 raft_consensus.cc:348] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.444608 2624 raft_consensus.cc:374] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:01.444604 2623 tablet_bootstrap.cc:654] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:01.444629 2624 raft_consensus.cc:729] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b11085737f74876ab2499caefa8e665, State: Initialized, Role: FOLLOWER
I20260927 03:23:01.444603 2627 consensus_queue.cc:260] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.444676 2624 consensus_queue.cc:260] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.444772 2624 ts_tablet_manager.cc:1428] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:01.444805 2629 tablet_replica.cc:439] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } }
I20260927 03:23:01.444801 2631 tablet_replica.cc:439] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } }
I20260927 03:23:01.444869 2629 tablet_replica.cc:442] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:23:01.444890 2631 tablet_replica.cc:442] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:23:01.444934 2627 ts_tablet_manager.cc:1428] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:01.445670 2623 tablet_bootstrap.cc:492] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9: No bootstrap required, opened a new log
I20260927 03:23:01.445714 2623 ts_tablet_manager.cc:1397] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:23:01.445842 2623 raft_consensus.cc:348] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.445894 2623 raft_consensus.cc:374] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:01.445911 2623 raft_consensus.cc:729] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b23280a2a94ff6b7c3be2b8436b2c9, State: Initialized, Role: FOLLOWER
I20260927 03:23:01.445962 2623 consensus_queue.cc:260] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.446041 2623 ts_tablet_manager.cc:1428] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:01.446105 2628 tablet_replica.cc:439] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } }
I20260927 03:23:01.446199 2628 tablet_replica.cc:442] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 03:23:01.463553 2604 tablet.cc:2369] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:23:01.531507 2628 raft_consensus.cc:482] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:01.531625 2628 raft_consensus.cc:504] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.532104 2628 leader_election.cc:288] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cccd6746a5548a3af3923170e8e0938 (127.0.98.1:43593), 7b11085737f74876ab2499caefa8e665 (127.0.98.2:36457)
I20260927 03:23:01.535830 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.532178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43593 (local address 127.0.98.3:57231)
0927 03:23:01.532393 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.532401 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.532448 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.532551 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.532552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.532553 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:01.532604 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.532605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.533424 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.533424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.534695 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.534699 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.535535 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.535536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.535628 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.535634 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.535694 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.535737 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":41,"thread_start_us":92,"threads_started":1}
I20260927 03:23:01.535854 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.532283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57231 (local address 127.0.98.1:43593)
0927 03:23:01.532331 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.532333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.532456 (+ 123us) server_negotiation.cc:407] Connection header received
0927 03:23:01.532486 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.532487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.532495 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.532538 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:01.532658 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.533353 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.534802 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.535425 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.535519 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.535544 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.535733 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.535787 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:23:01.535882 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.532305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42139 (local address 127.0.98.2:36457)
0927 03:23:01.532350 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.532352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.532379 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:23:01.532402 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.532403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.532413 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.532459 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:01.532619 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.533356 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.534801 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.535490 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.535582 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.535593 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.535784 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.535826 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:23:01.535912 2496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.532162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36457 (local address 127.0.98.3:42139)
0927 03:23:01.532273 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.532286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.532333 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.532474 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.532475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.532477 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:01.532554 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.532556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.533423 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.533424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.534695 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.534699 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.535582 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.535583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.535704 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.535707 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.535754 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.535799 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":40}
I20260927 03:23:01.536346 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf53425b9a94f7596cd0616e145fb6b" candidate_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b11085737f74876ab2499caefa8e665" is_pre_election: true
I20260927 03:23:01.536473 2417 raft_consensus.cc:2454] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b23280a2a94ff6b7c3be2b8436b2c9 in term 0.
I20260927 03:23:01.536638 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf53425b9a94f7596cd0616e145fb6b" candidate_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cccd6746a5548a3af3923170e8e0938" is_pre_election: true
I20260927 03:23:01.536665 2490 leader_election.cc:302] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b23280a2a94ff6b7c3be2b8436b2c9, 7b11085737f74876ab2499caefa8e665; no voters:
I20260927 03:23:01.536732 2277 raft_consensus.cc:2454] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b23280a2a94ff6b7c3be2b8436b2c9 in term 0.
I20260927 03:23:01.536880 2628 raft_consensus.cc:2781] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:23:01.536926 2628 raft_consensus.cc:482] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:01.536939 2628 raft_consensus.cc:3037] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:01.537619 2628 raft_consensus.cc:504] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.537732 2628 leader_election.cc:288] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [CANDIDATE]: Term 1 election: Requested vote from peers 6cccd6746a5548a3af3923170e8e0938 (127.0.98.1:43593), 7b11085737f74876ab2499caefa8e665 (127.0.98.2:36457)
I20260927 03:23:01.537878 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf53425b9a94f7596cd0616e145fb6b" candidate_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b11085737f74876ab2499caefa8e665"
I20260927 03:23:01.537951 2417 raft_consensus.cc:3037] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:01.537940 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf53425b9a94f7596cd0616e145fb6b" candidate_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cccd6746a5548a3af3923170e8e0938"
I20260927 03:23:01.538007 2277 raft_consensus.cc:3037] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:01.538576 2277 raft_consensus.cc:2454] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b23280a2a94ff6b7c3be2b8436b2c9 in term 1.
I20260927 03:23:01.538583 2417 raft_consensus.cc:2454] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b23280a2a94ff6b7c3be2b8436b2c9 in term 1.
I20260927 03:23:01.538736 2488 leader_election.cc:302] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b23280a2a94ff6b7c3be2b8436b2c9, 6cccd6746a5548a3af3923170e8e0938; no voters:
I20260927 03:23:01.538837 2628 raft_consensus.cc:2781] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:01.538971 2628 raft_consensus.cc:686] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 LEADER]: Becoming Leader. State: Replica: 39b23280a2a94ff6b7c3be2b8436b2c9, State: Running, Role: LEADER
I20260927 03:23:01.539068 2628 consensus_queue.cc:237] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.539464 2645 tablet_replica.cc:439] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9: TxnStatusTablet state changed. Reason: New leader 39b23280a2a94ff6b7c3be2b8436b2c9. Latest consensus state: current_term: 1 leader_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } }
I20260927 03:23:01.539527 2645 tablet_replica.cc:442] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:23:01.539681 2647 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 03:23:01.539777 2647 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 03:23:01.539824 2119 catalog_manager.cc:5495] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39b23280a2a94ff6b7c3be2b8436b2c9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } health_report { overall_health: HEALTHY } } }
I20260927 03:23:01.564718 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.561957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44631 (local address 127.0.0.1:36338)
0927 03:23:01.562031 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.562043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.562093 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.562249 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.562250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.562251 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:23:01.562303 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.562306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.563011 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.563012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.563673 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.563677 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.564176 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.564177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.564252 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.564272 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:23:01.564539 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:23:01.564554 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.564597 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.564649 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 03:23:01.564716 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.562032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36338 (local address 127.0.98.3:44631)
0927 03:23:01.562100 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.562102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.562111 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:23:01.562166 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.562167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.562187 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.562242 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:01.562363 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.562935 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.563762 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.564056 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.564201 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.564372 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:01.564483 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:01.564554 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.564602 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.564631 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 03:23:01.566624 2277 raft_consensus.cc:1261] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938 [term 1 FOLLOWER]: Refusing update from remote peer 39b23280a2a94ff6b7c3be2b8436b2c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:01.566658 2417 raft_consensus.cc:1261] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Refusing update from remote peer 39b23280a2a94ff6b7c3be2b8436b2c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:01.567065 2645 consensus_queue.cc:1035] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:01.567198 2628 consensus_queue.cc:1035] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:01.568331 2631 tablet_replica.cc:439] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938: TxnStatusTablet state changed. Reason: New leader 39b23280a2a94ff6b7c3be2b8436b2c9. Latest consensus state: current_term: 1 leader_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } }
I20260927 03:23:01.568343 2629 tablet_replica.cc:439] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665: TxnStatusTablet state changed. Reason: New leader 39b23280a2a94ff6b7c3be2b8436b2c9. Latest consensus state: current_term: 1 leader_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } }
I20260927 03:23:01.568418 2629 tablet_replica.cc:442] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:23:01.568419 2631 tablet_replica.cc:442] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:23:01.569515 2628 tablet_replica.cc:439] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } }
I20260927 03:23:01.569612 2628 tablet_replica.cc:442] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:23:01.569710 2631 tablet_replica.cc:439] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } }
I20260927 03:23:01.569797 2631 tablet_replica.cc:442] T faf53425b9a94f7596cd0616e145fb6b P 6cccd6746a5548a3af3923170e8e0938: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:23:01.569867 2650 mvcc.cc:205] Tried to move back new op lower bound from 7333803546894434304 to 7333803546784673792. Current Snapshot: MvccSnapshot[applied={T|T < 7333803546894434304}]
I20260927 03:23:01.570026 2645 tablet_replica.cc:439] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } }
I20260927 03:23:01.570192 2645 tablet_replica.cc:442] T faf53425b9a94f7596cd0616e145fb6b P 39b23280a2a94ff6b7c3be2b8436b2c9: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:23:01.570595 2629 tablet_replica.cc:439] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } } }
I20260927 03:23:01.570633 2652 mvcc.cc:205] Tried to move back new op lower bound from 7333803546894434304 to 7333803546784673792. Current Snapshot: MvccSnapshot[applied={T|T < 7333803546894434304 or (T in {7333803546894434304})}]
I20260927 03:23:01.570676 2629 tablet_replica.cc:442] T faf53425b9a94f7596cd0616e145fb6b P 7b11085737f74876ab2499caefa8e665: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:23:01.571005 2651 mvcc.cc:205] Tried to move back new op lower bound from 7333803546894434304 to 7333803546784673792. Current Snapshot: MvccSnapshot[applied={T|T < 7333803546894434304}]
I20260927 03:23:01.575074 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.572731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.0.98.2:36457)
0927 03:23:01.572771 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.572773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.572781 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:01.572812 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.572813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.572831 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.572893 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:01.573000 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.573481 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.574281 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.574507 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.574561 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.574704 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:01.574854 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:01.574929 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.574961 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.574999 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:23:01.580205 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.577689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41285 (local address 127.0.98.3:44631)
0927 03:23:01.577748 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.577750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.577758 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:01.577784 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.577785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.577794 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.577835 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:01.577984 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.578424 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.579559 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.579755 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.579821 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.580019 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:01.580047 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:01.580078 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.580133 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.580148 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:23:01.580257 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.577483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44631 (local address 127.0.98.2:41285)
0927 03:23:01.577563 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.577575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.577619 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.577859 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.577860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.577862 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:23:01.577925 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.577926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.578494 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.578494 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.579462 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.579464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.579834 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.579835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.579901 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.579918 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:23:01.580084 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:23:01.580086 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.580131 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.580180 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 03:23:01.582576 2277 raft_consensus.cc:1261] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938 [term 1 FOLLOWER]: Refusing update from remote peer 7b11085737f74876ab2499caefa8e665: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:01.582680 2550 raft_consensus.cc:1261] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Refusing update from remote peer 7b11085737f74876ab2499caefa8e665: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:01.582863 2633 consensus_queue.cc:1035] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:01.582989 2629 consensus_queue.cc:1035] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:01.591246 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.588795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36457 (local address 127.0.98.3:43055)
0927 03:23:01.588850 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.588862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.588902 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.589053 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.589054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.589055 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:23:01.589130 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.589132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.589769 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.589770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.590463 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.590466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.590804 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.590804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.590912 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.590926 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:23:01.591088 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:23:01.591090 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.591136 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.591179 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 03:23:01.591240 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.588853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43055 (local address 127.0.98.2:36457)
0927 03:23:01.588886 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.588888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.588896 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:01.588957 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.588957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.588970 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.589008 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:01.589210 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.589692 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.590539 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.590746 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.590800 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.590994 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:01.591040 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:01.591082 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.591138 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.591160 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:23:01.599246 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:01.599357 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:01.600101 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b: Sending DeleteTablet for 3 replicas of tablet 0d3fe8b88a664af2ae1594fe0f301b2a
I20260927 03:23:01.600397 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3fe8b88a664af2ae1594fe0f301b2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49092
I20260927 03:23:01.600694 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3fe8b88a664af2ae1594fe0f301b2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44080
I20260927 03:23:01.600973 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3fe8b88a664af2ae1594fe0f301b2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36332
I20260927 03:23:01.601186 2671 tablet_replica.cc:330] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9: stopping tablet replica
I20260927 03:23:01.601227 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:23:01.601370 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:23:01.601485 2671 raft_consensus.cc:2229] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:01.601598 2671 raft_consensus.cc:2258] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:01.602166 2671 ts_tablet_manager.cc:1905] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:01.602208 2669 tablet_replica.cc:330] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665: stopping tablet replica
I20260927 03:23:01.602299 2669 raft_consensus.cc:2229] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:23:01.602399 2669 raft_consensus.cc:2258] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:01.602557 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 75334053b56b4ab1b50348a78c1edb02 (DEFAULT_TABLE table=test-table [id=8c9056ed5173456c926ebae96eed7475]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:01.602698 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75334053b56b4ab1b50348a78c1edb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:01.602790 2669 ts_tablet_manager.cc:1905] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:01.604077 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 75334053b56b4ab1b50348a78c1edb02 (DEFAULT_TABLE table=test-table [id=8c9056ed5173456c926ebae96eed7475]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:01.604231 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75334053b56b4ab1b50348a78c1edb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:01.604477 2669 ts_tablet_manager.cc:1918] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:23:01.604539 2669 log.cc:1196] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/0d3fe8b88a664af2ae1594fe0f301b2a
I20260927 03:23:01.604750 2669 ts_tablet_manager.cc:1939] T 0d3fe8b88a664af2ae1594fe0f301b2a P 7b11085737f74876ab2499caefa8e665: Deleting consensus metadata
I20260927 03:23:01.604924 2670 tablet_replica.cc:330] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938: stopping tablet replica
I20260927 03:23:01.605038 2670 raft_consensus.cc:2229] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:01.605098 2670 raft_consensus.cc:2258] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:01.605423 2627 tablet_bootstrap.cc:492] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938: Bootstrap starting.
I20260927 03:23:01.605525 2670 ts_tablet_manager.cc:1905] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:01.605821 2627 tablet_bootstrap.cc:654] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:01.606146 2624 tablet_bootstrap.cc:492] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665: Bootstrap starting.
I20260927 03:23:01.606511 2624 tablet_bootstrap.cc:654] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:01.607031 2106 catalog_manager.cc:4841] TS 7b11085737f74876ab2499caefa8e665 (127.0.98.2:36457): tablet 0d3fe8b88a664af2ae1594fe0f301b2a (table test-table [id=13cffc94590e4c81af74d22ad2f537bb]) successfully deleted
I20260927 03:23:01.607117 2670 ts_tablet_manager.cc:1918] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:23:01.607185 2670 log.cc:1196] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/wal/wals/0d3fe8b88a664af2ae1594fe0f301b2a
I20260927 03:23:01.607511 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 75334053b56b4ab1b50348a78c1edb02 (DEFAULT_TABLE table=test-table [id=8c9056ed5173456c926ebae96eed7475]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:01.607555 2624 tablet_bootstrap.cc:492] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665: No bootstrap required, opened a new log
I20260927 03:23:01.607595 2624 ts_tablet_manager.cc:1397] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:01.607692 2670 ts_tablet_manager.cc:1939] T 0d3fe8b88a664af2ae1594fe0f301b2a P 6cccd6746a5548a3af3923170e8e0938: Deleting consensus metadata
I20260927 03:23:01.607741 2624 raft_consensus.cc:348] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.607820 2624 raft_consensus.cc:374] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:01.607848 2624 raft_consensus.cc:729] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b11085737f74876ab2499caefa8e665, State: Initialized, Role: FOLLOWER
I20260927 03:23:01.607888 2624 consensus_queue.cc:260] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.607981 2624 ts_tablet_manager.cc:1428] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:01.608104 2104 catalog_manager.cc:4841] TS 6cccd6746a5548a3af3923170e8e0938 (127.0.98.1:43593): tablet 0d3fe8b88a664af2ae1594fe0f301b2a (table test-table [id=13cffc94590e4c81af74d22ad2f537bb]) successfully deleted
I20260927 03:23:01.608318 2627 tablet_bootstrap.cc:492] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938: No bootstrap required, opened a new log
I20260927 03:23:01.608373 2627 ts_tablet_manager.cc:1397] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 03:23:01.608508 2627 raft_consensus.cc:348] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.608584 2627 raft_consensus.cc:374] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:01.608611 2627 raft_consensus.cc:729] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cccd6746a5548a3af3923170e8e0938, State: Initialized, Role: FOLLOWER
I20260927 03:23:01.608659 2627 consensus_queue.cc:260] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.608754 2627 ts_tablet_manager.cc:1428] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:01.611002 2671 ts_tablet_manager.cc:1918] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:23:01.611002 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75334053b56b4ab1b50348a78c1edb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:01.611163 2671 log.cc:1196] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/0d3fe8b88a664af2ae1594fe0f301b2a
I20260927 03:23:01.611429 2671 ts_tablet_manager.cc:1939] T 0d3fe8b88a664af2ae1594fe0f301b2a P 39b23280a2a94ff6b7c3be2b8436b2c9: Deleting consensus metadata
I20260927 03:23:01.611972 2105 catalog_manager.cc:4841] TS 39b23280a2a94ff6b7c3be2b8436b2c9 (127.0.98.3:44631): tablet 0d3fe8b88a664af2ae1594fe0f301b2a (table test-table [id=13cffc94590e4c81af74d22ad2f537bb]) successfully deleted
I20260927 03:23:01.612433 2623 tablet_bootstrap.cc:492] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9: Bootstrap starting.
I20260927 03:23:01.612751 2623 tablet_bootstrap.cc:654] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:01.613292 2623 tablet_bootstrap.cc:492] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9: No bootstrap required, opened a new log
I20260927 03:23:01.613333 2623 ts_tablet_manager.cc:1397] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:01.613466 2623 raft_consensus.cc:348] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.613521 2623 raft_consensus.cc:374] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:01.613538 2623 raft_consensus.cc:729] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b23280a2a94ff6b7c3be2b8436b2c9, State: Initialized, Role: FOLLOWER
I20260927 03:23:01.613591 2623 consensus_queue.cc:260] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.613675 2623 ts_tablet_manager.cc:1428] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:01.868602 2631 raft_consensus.cc:482] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:01.868782 2631 raft_consensus.cc:504] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.869143 2631 leader_election.cc:288] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b11085737f74876ab2499caefa8e665 (127.0.98.2:36457), 39b23280a2a94ff6b7c3be2b8436b2c9 (127.0.98.3:44631)
I20260927 03:23:01.872550 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.869503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40743 (local address 127.0.98.2:36457)
0927 03:23:01.869561 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.869564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.869574 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:01.869681 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.869682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.869692 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.869722 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:01.869951 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.870493 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.871714 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.872183 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.872236 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.872246 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.872459 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.872494 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:23:01.872651 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.869347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36457 (local address 127.0.98.1:40743)
0927 03:23:01.869514 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.869525 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.869581 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.869791 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.869792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.869794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:01.869873 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.869875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.870653 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.870654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.871616 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.871619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.872268 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.872270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.872349 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.872355 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.872422 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.872463 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20260927 03:23:01.872895 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.869349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44631 (local address 127.0.98.1:60051)
0927 03:23:01.869536 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.869547 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.869605 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.870126 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.870127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.870129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:01.870195 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.870197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.871111 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.871112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.871962 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.871965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.872690 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.872691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.872759 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.872763 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.872813 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.872844 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":93,"threads_started":1}
I20260927 03:23:01.872933 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.869461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60051 (local address 127.0.98.3:44631)
0927 03:23:01.869998 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.870000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.870008 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:01.870033 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.870034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.870044 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.870108 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:01.870349 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.870989 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.872084 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.872562 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.872629 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.872638 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.872833 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.872871 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504}
I20260927 03:23:01.873008 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75334053b56b4ab1b50348a78c1edb02" candidate_uuid: "6cccd6746a5548a3af3923170e8e0938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b11085737f74876ab2499caefa8e665" is_pre_election: true
I20260927 03:23:01.873107 2417 raft_consensus.cc:2454] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cccd6746a5548a3af3923170e8e0938 in term 0.
I20260927 03:23:01.873090 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75334053b56b4ab1b50348a78c1edb02" candidate_uuid: "6cccd6746a5548a3af3923170e8e0938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" is_pre_election: true
I20260927 03:23:01.873191 2550 raft_consensus.cc:2454] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cccd6746a5548a3af3923170e8e0938 in term 0.
I20260927 03:23:01.873279 2210 leader_election.cc:302] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cccd6746a5548a3af3923170e8e0938, 7b11085737f74876ab2499caefa8e665; no voters:
I20260927 03:23:01.873418 2631 raft_consensus.cc:2781] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:23:01.873461 2631 raft_consensus.cc:482] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:01.873479 2631 raft_consensus.cc:3037] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:01.874222 2631 raft_consensus.cc:504] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.874346 2631 leader_election.cc:288] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [CANDIDATE]: Term 1 election: Requested vote from peers 7b11085737f74876ab2499caefa8e665 (127.0.98.2:36457), 39b23280a2a94ff6b7c3be2b8436b2c9 (127.0.98.3:44631)
I20260927 03:23:01.874509 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75334053b56b4ab1b50348a78c1edb02" candidate_uuid: "6cccd6746a5548a3af3923170e8e0938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b11085737f74876ab2499caefa8e665"
I20260927 03:23:01.874578 2417 raft_consensus.cc:3037] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:01.874604 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75334053b56b4ab1b50348a78c1edb02" candidate_uuid: "6cccd6746a5548a3af3923170e8e0938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9"
I20260927 03:23:01.874689 2550 raft_consensus.cc:3037] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:01.875217 2417 raft_consensus.cc:2454] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cccd6746a5548a3af3923170e8e0938 in term 1.
I20260927 03:23:01.875378 2210 leader_election.cc:302] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cccd6746a5548a3af3923170e8e0938, 7b11085737f74876ab2499caefa8e665; no voters:
I20260927 03:23:01.875389 2550 raft_consensus.cc:2454] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cccd6746a5548a3af3923170e8e0938 in term 1.
I20260927 03:23:01.875480 2631 raft_consensus.cc:2781] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:01.875646 2631 raft_consensus.cc:686] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [term 1 LEADER]: Becoming Leader. State: Replica: 6cccd6746a5548a3af3923170e8e0938, State: Running, Role: LEADER
I20260927 03:23:01.875725 2631 consensus_queue.cc:237] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:01.876439 2119 catalog_manager.cc:5495] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cccd6746a5548a3af3923170e8e0938 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6cccd6746a5548a3af3923170e8e0938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } health_report { overall_health: UNKNOWN } } }
I20260927 03:23:01.895888 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.893487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.0.98.1:43593)
0927 03:23:01.893529 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.893531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.893554 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:23:01.893629 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.893630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.893649 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.893707 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:01.893802 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.894309 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.895142 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.895356 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.895421 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.895564 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:01.895670 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:01.895722 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.895806 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.895823 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:23:01.901095 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.898770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47277 (local address 127.0.98.3:44631)
0927 03:23:01.898862 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.898864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.898932 (+ 68us) server_negotiation.cc:407] Connection header received
0927 03:23:01.898996 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.898997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.899008 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.899051 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:01.899209 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.899752 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.900488 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.900675 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.900728 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.900874 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:01.900904 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:01.900944 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.900997 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.901014 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 03:23:01.901131 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.898604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44631 (local address 127.0.98.1:47277)
0927 03:23:01.898869 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.898880 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.898939 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.899057 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.899057 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.899059 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:23:01.899133 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.899135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.899824 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.899825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.900417 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.900418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.900731 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.900732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.900795 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.900808 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:23:01.900948 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:23:01.900950 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.900986 (+ 36us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.901032 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":152,"threads_started":1}
I20260927 03:23:01.903343 2550 raft_consensus.cc:1261] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Refusing update from remote peer 6cccd6746a5548a3af3923170e8e0938: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:01.903491 2417 raft_consensus.cc:1261] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Refusing update from remote peer 6cccd6746a5548a3af3923170e8e0938: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:01.903605 2677 consensus_queue.cc:1035] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:01.903743 2631 consensus_queue.cc:1035] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:01.910993 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.908677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55949 (local address 127.0.98.1:43593)
0927 03:23:01.908721 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:01.908723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:01.908814 (+ 91us) server_negotiation.cc:407] Connection header received
0927 03:23:01.908874 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:01.908875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:01.908884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:01.908933 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:01.909070 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.909600 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.910303 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.910557 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.910611 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.910735 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:01.910770 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:01.910811 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:01.910890 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 03:23:01.910913 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:23:01.911022 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:01.908703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43593 (local address 127.0.98.3:55949)
0927 03:23:01.908754 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:01.908764 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:01.908820 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:01.908937 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:01.908938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:01.908939 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:23:01.908983 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.908985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.909672 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.909673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.910213 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:01.910215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:01.910613 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:01.910614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:01.910669 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:01.910673 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:23:01.910837 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:23:01.910839 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:23:01.910886 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:23:01.910933 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:23:01.911427 2652 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 03:23:03.431461 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:03.431573 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:03.432386 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b: Sending DeleteTablet for 3 replicas of tablet 75334053b56b4ab1b50348a78c1edb02
I20260927 03:23:03.432667 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 75334053b56b4ab1b50348a78c1edb02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49092
I20260927 03:23:03.432679 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 75334053b56b4ab1b50348a78c1edb02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44080
I20260927 03:23:03.432693 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 75334053b56b4ab1b50348a78c1edb02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36332
I20260927 03:23:03.432883 2711 tablet_replica.cc:330] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938: stopping tablet replica
I20260927 03:23:03.432938 2711 raft_consensus.cc:2229] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:23:03.433175 2711 raft_consensus.cc:2258] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:03.433295 2712 tablet_replica.cc:330] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665: stopping tablet replica
I20260927 03:23:03.433353 2712 raft_consensus.cc:2229] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:03.433393 2712 raft_consensus.cc:2258] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:03.433575 2711 ts_tablet_manager.cc:1905] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:03.434998 2713 tablet_replica.cc:330] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9: stopping tablet replica
I20260927 03:23:03.435061 2713 raft_consensus.cc:2229] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:03.435119 2713 raft_consensus.cc:2258] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:03.435312 2711 ts_tablet_manager.cc:1918] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:23:03.435370 2711 log.cc:1196] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/wal/wals/75334053b56b4ab1b50348a78c1edb02
I20260927 03:23:03.435379 2713 ts_tablet_manager.cc:1905] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:03.435565 2711 ts_tablet_manager.cc:1939] T 75334053b56b4ab1b50348a78c1edb02 P 6cccd6746a5548a3af3923170e8e0938: Deleting consensus metadata
I20260927 03:23:03.435806 2712 ts_tablet_manager.cc:1905] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:03.435935 2104 catalog_manager.cc:4841] TS 6cccd6746a5548a3af3923170e8e0938 (127.0.98.1:43593): tablet 75334053b56b4ab1b50348a78c1edb02 (table test-table [id=8c9056ed5173456c926ebae96eed7475]) successfully deleted
I20260927 03:23:03.437548 2712 ts_tablet_manager.cc:1918] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:23:03.437608 2712 log.cc:1196] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/75334053b56b4ab1b50348a78c1edb02
I20260927 03:23:03.437795 2712 ts_tablet_manager.cc:1939] T 75334053b56b4ab1b50348a78c1edb02 P 7b11085737f74876ab2499caefa8e665: Deleting consensus metadata
I20260927 03:23:03.438120 2106 catalog_manager.cc:4841] TS 7b11085737f74876ab2499caefa8e665 (127.0.98.2:36457): tablet 75334053b56b4ab1b50348a78c1edb02 (table test-table [id=8c9056ed5173456c926ebae96eed7475]) successfully deleted
I20260927 03:23:03.438596 2713 ts_tablet_manager.cc:1918] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:23:03.438645 2713 log.cc:1196] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/75334053b56b4ab1b50348a78c1edb02
I20260927 03:23:03.438834 2713 ts_tablet_manager.cc:1939] T 75334053b56b4ab1b50348a78c1edb02 P 39b23280a2a94ff6b7c3be2b8436b2c9: Deleting consensus metadata
I20260927 03:23:03.439301 2105 catalog_manager.cc:4841] TS 39b23280a2a94ff6b7c3be2b8436b2c9 (127.0.98.3:44631): tablet 75334053b56b4ab1b50348a78c1edb02 (table test-table [id=8c9056ed5173456c926ebae96eed7475]) successfully deleted
Sep 27 03:23:03 dist-test-slave-gw2v krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479383, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:23:03 dist-test-slave-gw2v krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479383, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:03.456710 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:03.448889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.0.98.62:44377)
0927 03:23:03.449021 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:03.449023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:03.449105 (+ 82us) server_negotiation.cc:407] Connection header received
0927 03:23:03.449158 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:03.449159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:03.449173 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:03.449258 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:03.449874 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:03.450560 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:03.451409 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:03.451640 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:03.451695 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:03.454009 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:03.454010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:03.454011 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:03.454020 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:03.455535 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:03.455866 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:03.455867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:03.455867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:03.455923 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:03.456133 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:03.456133 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:03.456134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:03.456242 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:03.456315 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:03.456564 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 03:23:03.456634 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20260927 03:23:03.458180 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:23:03.458328 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:23:03.459560 2257 tablet_service.cc:1467] Processing CreateTablet for tablet ff92046fed714ed092bd88d4250418de (DEFAULT_TABLE table=test-table [id=57f7eaa08a0746d28a8201d0130d50cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:03.459584 2537 tablet_service.cc:1467] Processing CreateTablet for tablet ff92046fed714ed092bd88d4250418de (DEFAULT_TABLE table=test-table [id=57f7eaa08a0746d28a8201d0130d50cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:03.459705 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff92046fed714ed092bd88d4250418de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:03.459707 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff92046fed714ed092bd88d4250418de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:03.461041 2722 tablet_bootstrap.cc:492] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9: Bootstrap starting.
I20260927 03:23:03.461086 2397 tablet_service.cc:1467] Processing CreateTablet for tablet ff92046fed714ed092bd88d4250418de (DEFAULT_TABLE table=test-table [id=57f7eaa08a0746d28a8201d0130d50cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:03.461200 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff92046fed714ed092bd88d4250418de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:03.461404 2722 tablet_bootstrap.cc:654] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:03.461957 2722 tablet_bootstrap.cc:492] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9: No bootstrap required, opened a new log
I20260927 03:23:03.462000 2722 ts_tablet_manager.cc:1397] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:03.462122 2722 raft_consensus.cc:348] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.462178 2722 raft_consensus.cc:374] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:03.462196 2722 raft_consensus.cc:729] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b23280a2a94ff6b7c3be2b8436b2c9, State: Initialized, Role: FOLLOWER
I20260927 03:23:03.462241 2722 consensus_queue.cc:260] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.462363 2725 tablet_bootstrap.cc:492] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665: Bootstrap starting.
I20260927 03:23:03.462376 2722 ts_tablet_manager.cc:1428] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:03.462590 2723 tablet_bootstrap.cc:492] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938: Bootstrap starting.
I20260927 03:23:03.462978 2725 tablet_bootstrap.cc:654] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:03.463074 2723 tablet_bootstrap.cc:654] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:03.463582 2725 tablet_bootstrap.cc:492] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665: No bootstrap required, opened a new log
I20260927 03:23:03.463585 2723 tablet_bootstrap.cc:492] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938: No bootstrap required, opened a new log
I20260927 03:23:03.463624 2725 ts_tablet_manager.cc:1397] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:23:03.463627 2723 ts_tablet_manager.cc:1397] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:03.463702 2725 raft_consensus.cc:348] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.463752 2725 raft_consensus.cc:374] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:03.463765 2725 raft_consensus.cc:729] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b11085737f74876ab2499caefa8e665, State: Initialized, Role: FOLLOWER
I20260927 03:23:03.463744 2723 raft_consensus.cc:348] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.463815 2723 raft_consensus.cc:374] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:03.463801 2725 consensus_queue.cc:260] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.463837 2723 raft_consensus.cc:729] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cccd6746a5548a3af3923170e8e0938, State: Initialized, Role: FOLLOWER
I20260927 03:23:03.463881 2723 consensus_queue.cc:260] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.463946 2725 ts_tablet_manager.cc:1428] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:03.463982 2723 ts_tablet_manager.cc:1428] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:03.465848 2706 raft_consensus.cc:482] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:03.465903 2706 raft_consensus.cc:504] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.466049 2706 leader_election.cc:288] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b11085737f74876ab2499caefa8e665 (127.0.98.2:36457), 39b23280a2a94ff6b7c3be2b8436b2c9 (127.0.98.3:44631)
I20260927 03:23:03.466193 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff92046fed714ed092bd88d4250418de" candidate_uuid: "6cccd6746a5548a3af3923170e8e0938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b11085737f74876ab2499caefa8e665" is_pre_election: true
I20260927 03:23:03.466199 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff92046fed714ed092bd88d4250418de" candidate_uuid: "6cccd6746a5548a3af3923170e8e0938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" is_pre_election: true
I20260927 03:23:03.466266 2417 raft_consensus.cc:2454] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cccd6746a5548a3af3923170e8e0938 in term 0.
I20260927 03:23:03.466266 2550 raft_consensus.cc:2454] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cccd6746a5548a3af3923170e8e0938 in term 0.
I20260927 03:23:03.466393 2209 leader_election.cc:302] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b23280a2a94ff6b7c3be2b8436b2c9, 6cccd6746a5548a3af3923170e8e0938; no voters:
I20260927 03:23:03.466516 2706 raft_consensus.cc:2781] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:23:03.466574 2706 raft_consensus.cc:482] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:03.466588 2706 raft_consensus.cc:3037] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:03.467235 2706 raft_consensus.cc:504] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.467363 2706 leader_election.cc:288] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [CANDIDATE]: Term 1 election: Requested vote from peers 7b11085737f74876ab2499caefa8e665 (127.0.98.2:36457), 39b23280a2a94ff6b7c3be2b8436b2c9 (127.0.98.3:44631)
I20260927 03:23:03.467516 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff92046fed714ed092bd88d4250418de" candidate_uuid: "6cccd6746a5548a3af3923170e8e0938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b11085737f74876ab2499caefa8e665"
I20260927 03:23:03.467528 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff92046fed714ed092bd88d4250418de" candidate_uuid: "6cccd6746a5548a3af3923170e8e0938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9"
I20260927 03:23:03.467595 2417 raft_consensus.cc:3037] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:03.467595 2550 raft_consensus.cc:3037] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:03.468314 2417 raft_consensus.cc:2454] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cccd6746a5548a3af3923170e8e0938 in term 1.
I20260927 03:23:03.468314 2550 raft_consensus.cc:2454] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cccd6746a5548a3af3923170e8e0938 in term 1.
I20260927 03:23:03.468483 2209 leader_election.cc:302] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b23280a2a94ff6b7c3be2b8436b2c9, 6cccd6746a5548a3af3923170e8e0938; no voters:
I20260927 03:23:03.468564 2706 raft_consensus.cc:2781] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:03.468626 2706 raft_consensus.cc:686] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [term 1 LEADER]: Becoming Leader. State: Replica: 6cccd6746a5548a3af3923170e8e0938, State: Running, Role: LEADER
I20260927 03:23:03.468675 2706 consensus_queue.cc:237] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.469197 2118 catalog_manager.cc:5495] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cccd6746a5548a3af3923170e8e0938 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6cccd6746a5548a3af3923170e8e0938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } health_report { overall_health: UNKNOWN } } }
I20260927 03:23:03.477629 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:03.475179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44098 (local address 127.0.98.1:43593)
0927 03:23:03.475319 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:03.475321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:03.475329 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:03.475349 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:03.475350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:03.475359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:03.475487 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:03.475584 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:03.476080 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:03.476949 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:03.477168 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:03.477225 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:03.477391 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:03.477459 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:03.477496 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:03.477547 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 03:23:03.477565 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260927 03:23:03.479607 2417 raft_consensus.cc:1261] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Refusing update from remote peer 6cccd6746a5548a3af3923170e8e0938: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:03.479621 2550 raft_consensus.cc:1261] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Refusing update from remote peer 6cccd6746a5548a3af3923170e8e0938: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:03.479803 2701 consensus_queue.cc:1035] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:03.479918 2706 consensus_queue.cc:1035] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:03.488919 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:03.489009 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:03.489584 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b: Sending DeleteTablet for 3 replicas of tablet ff92046fed714ed092bd88d4250418de
I20260927 03:23:03.489815 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet ff92046fed714ed092bd88d4250418de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36332
I20260927 03:23:03.489822 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet ff92046fed714ed092bd88d4250418de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44080
I20260927 03:23:03.489936 2713 tablet_replica.cc:330] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9: stopping tablet replica
I20260927 03:23:03.489926 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet ff92046fed714ed092bd88d4250418de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49092
I20260927 03:23:03.490000 2713 raft_consensus.cc:2229] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:03.490038 2713 raft_consensus.cc:2258] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:03.490053 2712 tablet_replica.cc:330] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665: stopping tablet replica
I20260927 03:23:03.490115 2712 raft_consensus.cc:2229] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:03.490113 2711 tablet_replica.cc:330] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938: stopping tablet replica
I20260927 03:23:03.490159 2712 raft_consensus.cc:2258] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:03.490178 2711 raft_consensus.cc:2229] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:23:03.490231 2711 raft_consensus.cc:2258] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:03.490324 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 03:23:03.490407 2712 ts_tablet_manager.cc:1905] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 03:23:03.490455 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:23:03.490716 2711 ts_tablet_manager.cc:1905] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:03.492004 2713 ts_tablet_manager.cc:1905] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:03.492036 2712 ts_tablet_manager.cc:1918] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:23:03.492085 2712 log.cc:1196] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/ff92046fed714ed092bd88d4250418de
I20260927 03:23:03.492086 2711 ts_tablet_manager.cc:1918] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:23:03.492126 2711 log.cc:1196] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-0/wal/wals/ff92046fed714ed092bd88d4250418de
I20260927 03:23:03.492285 2712 ts_tablet_manager.cc:1939] T ff92046fed714ed092bd88d4250418de P 7b11085737f74876ab2499caefa8e665: Deleting consensus metadata
I20260927 03:23:03.492313 2711 ts_tablet_manager.cc:1939] T ff92046fed714ed092bd88d4250418de P 6cccd6746a5548a3af3923170e8e0938: Deleting consensus metadata
I20260927 03:23:03.492678 2106 catalog_manager.cc:4841] TS 7b11085737f74876ab2499caefa8e665 (127.0.98.2:36457): tablet ff92046fed714ed092bd88d4250418de (table test-table [id=57f7eaa08a0746d28a8201d0130d50cb]) successfully deleted
I20260927 03:23:03.492856 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba5c8ea58c242c7b9c893a826e7ab70 (DEFAULT_TABLE table=test-table [id=9a4ebff194af4e348ade30a336e5b4d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:03.492870 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba5c8ea58c242c7b9c893a826e7ab70 (DEFAULT_TABLE table=test-table [id=9a4ebff194af4e348ade30a336e5b4d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:03.492893 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba5c8ea58c242c7b9c893a826e7ab70 (DEFAULT_TABLE table=test-table [id=9a4ebff194af4e348ade30a336e5b4d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:03.492949 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba5c8ea58c242c7b9c893a826e7ab70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:03.492949 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba5c8ea58c242c7b9c893a826e7ab70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:03.492978 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba5c8ea58c242c7b9c893a826e7ab70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:03.493465 2713 ts_tablet_manager.cc:1918] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:23:03.493518 2713 log.cc:1196] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/ff92046fed714ed092bd88d4250418de
I20260927 03:23:03.493696 2713 ts_tablet_manager.cc:1939] T ff92046fed714ed092bd88d4250418de P 39b23280a2a94ff6b7c3be2b8436b2c9: Deleting consensus metadata
I20260927 03:23:03.494011 2722 tablet_bootstrap.cc:492] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9: Bootstrap starting.
I20260927 03:23:03.494012 2104 catalog_manager.cc:4841] TS 6cccd6746a5548a3af3923170e8e0938 (127.0.98.1:43593): tablet ff92046fed714ed092bd88d4250418de (table test-table [id=57f7eaa08a0746d28a8201d0130d50cb]) successfully deleted
I20260927 03:23:03.494059 2723 tablet_bootstrap.cc:492] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 6cccd6746a5548a3af3923170e8e0938: Bootstrap starting.
I20260927 03:23:03.494313 2723 tablet_bootstrap.cc:654] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 6cccd6746a5548a3af3923170e8e0938: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:03.494803 2722 tablet_bootstrap.cc:654] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:03.494855 2723 tablet_bootstrap.cc:492] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 6cccd6746a5548a3af3923170e8e0938: No bootstrap required, opened a new log
I20260927 03:23:03.494896 2723 ts_tablet_manager.cc:1397] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 6cccd6746a5548a3af3923170e8e0938: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:03.494915 2725 tablet_bootstrap.cc:492] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665: Bootstrap starting.
I20260927 03:23:03.495059 2723 raft_consensus.cc:348] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.495127 2723 raft_consensus.cc:374] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:03.495146 2723 raft_consensus.cc:729] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cccd6746a5548a3af3923170e8e0938, State: Initialized, Role: FOLLOWER
I20260927 03:23:03.495204 2725 tablet_bootstrap.cc:654] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:03.495195 2723 consensus_queue.cc:260] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 6cccd6746a5548a3af3923170e8e0938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.495288 2723 ts_tablet_manager.cc:1428] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 6cccd6746a5548a3af3923170e8e0938: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:03.495342 2722 tablet_bootstrap.cc:492] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9: No bootstrap required, opened a new log
I20260927 03:23:03.495375 2722 ts_tablet_manager.cc:1397] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:03.495467 2722 raft_consensus.cc:348] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.495525 2722 raft_consensus.cc:374] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:03.495543 2722 raft_consensus.cc:729] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b23280a2a94ff6b7c3be2b8436b2c9, State: Initialized, Role: FOLLOWER
I20260927 03:23:03.495569 2722 consensus_queue.cc:260] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.495653 2722 ts_tablet_manager.cc:1428] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:03.495711 2725 tablet_bootstrap.cc:492] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665: No bootstrap required, opened a new log
I20260927 03:23:03.495751 2725 ts_tablet_manager.cc:1397] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:23:03.495743 2105 catalog_manager.cc:4841] TS 39b23280a2a94ff6b7c3be2b8436b2c9 (127.0.98.3:44631): tablet ff92046fed714ed092bd88d4250418de (table test-table [id=57f7eaa08a0746d28a8201d0130d50cb]) successfully deleted
I20260927 03:23:03.495846 2725 raft_consensus.cc:348] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.495899 2725 raft_consensus.cc:374] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:03.495916 2725 raft_consensus.cc:729] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b11085737f74876ab2499caefa8e665, State: Initialized, Role: FOLLOWER
I20260927 03:23:03.495950 2725 consensus_queue.cc:260] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.496045 2725 ts_tablet_manager.cc:1428] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:03.800976 2728 raft_consensus.cc:482] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:03.801090 2728 raft_consensus.cc:504] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.801278 2728 leader_election.cc:288] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cccd6746a5548a3af3923170e8e0938 (127.0.98.1:43593), 39b23280a2a94ff6b7c3be2b8436b2c9 (127.0.98.3:44631)
I20260927 03:23:03.801538 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba5c8ea58c242c7b9c893a826e7ab70" candidate_uuid: "7b11085737f74876ab2499caefa8e665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" is_pre_election: true
I20260927 03:23:03.801530 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba5c8ea58c242c7b9c893a826e7ab70" candidate_uuid: "7b11085737f74876ab2499caefa8e665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cccd6746a5548a3af3923170e8e0938" is_pre_election: true
I20260927 03:23:03.801641 2277 raft_consensus.cc:2454] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b11085737f74876ab2499caefa8e665 in term 0.
I20260927 03:23:03.801643 2550 raft_consensus.cc:2454] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b11085737f74876ab2499caefa8e665 in term 0.
I20260927 03:23:03.801810 2348 leader_election.cc:302] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cccd6746a5548a3af3923170e8e0938, 7b11085737f74876ab2499caefa8e665; no voters:
I20260927 03:23:03.801916 2728 raft_consensus.cc:2781] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:23:03.801957 2728 raft_consensus.cc:482] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:03.801970 2728 raft_consensus.cc:3037] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:03.802688 2728 raft_consensus.cc:504] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.802801 2728 leader_election.cc:288] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [CANDIDATE]: Term 1 election: Requested vote from peers 6cccd6746a5548a3af3923170e8e0938 (127.0.98.1:43593), 39b23280a2a94ff6b7c3be2b8436b2c9 (127.0.98.3:44631)
I20260927 03:23:03.803001 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba5c8ea58c242c7b9c893a826e7ab70" candidate_uuid: "7b11085737f74876ab2499caefa8e665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cccd6746a5548a3af3923170e8e0938"
I20260927 03:23:03.803002 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba5c8ea58c242c7b9c893a826e7ab70" candidate_uuid: "7b11085737f74876ab2499caefa8e665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9"
I20260927 03:23:03.803081 2277 raft_consensus.cc:3037] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 6cccd6746a5548a3af3923170e8e0938 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:03.803081 2550 raft_consensus.cc:3037] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:03.803810 2550 raft_consensus.cc:2454] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b11085737f74876ab2499caefa8e665 in term 1.
I20260927 03:23:03.803810 2277 raft_consensus.cc:2454] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 6cccd6746a5548a3af3923170e8e0938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b11085737f74876ab2499caefa8e665 in term 1.
I20260927 03:23:03.803993 2348 leader_election.cc:302] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cccd6746a5548a3af3923170e8e0938, 7b11085737f74876ab2499caefa8e665; no voters:
I20260927 03:23:03.804095 2728 raft_consensus.cc:2781] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:03.804216 2728 raft_consensus.cc:686] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [term 1 LEADER]: Becoming Leader. State: Replica: 7b11085737f74876ab2499caefa8e665, State: Running, Role: LEADER
I20260927 03:23:03.804270 2728 consensus_queue.cc:237] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } }
I20260927 03:23:03.804888 2118 catalog_manager.cc:5495] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b11085737f74876ab2499caefa8e665 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7b11085737f74876ab2499caefa8e665" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b11085737f74876ab2499caefa8e665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 } health_report { overall_health: UNKNOWN } } }
I20260927 03:23:03.853664 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:03.851563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49122 (local address 127.0.98.2:36457)
0927 03:23:03.851722 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:03.851724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:03.851731 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:23:03.851749 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:03.851750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:03.851759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:03.851813 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:03.851925 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:03.852400 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:03.853098 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:03.853286 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:03.853327 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:03.853423 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:03.853487 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:03.853526 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:03.853562 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 03:23:03.853578 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260927 03:23:03.855877 2277 raft_consensus.cc:1261] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 6cccd6746a5548a3af3923170e8e0938 [term 1 FOLLOWER]: Refusing update from remote peer 7b11085737f74876ab2499caefa8e665: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:03.855969 2550 raft_consensus.cc:1261] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Refusing update from remote peer 7b11085737f74876ab2499caefa8e665: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:03.856169 2728 consensus_queue.cc:1035] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cccd6746a5548a3af3923170e8e0938" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:03.856293 2738 consensus_queue.cc:1035] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b23280a2a94ff6b7c3be2b8436b2c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:23:03.860357 2693 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 03:23:05.377559 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:05.377673 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:05.378533 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 594472a6516041b79e2f17a0cbf8ee9b: Sending DeleteTablet for 3 replicas of tablet 1ba5c8ea58c242c7b9c893a826e7ab70
I20260927 03:23:05.378785 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba5c8ea58c242c7b9c893a826e7ab70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44080
I20260927 03:23:05.378784 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba5c8ea58c242c7b9c893a826e7ab70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36332
I20260927 03:23:05.379004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 2192
I20260927 03:23:05.379453 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba5c8ea58c242c7b9c893a826e7ab70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49092
I20260927 03:23:05.379736 2767 tablet_replica.cc:330] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9: stopping tablet replica
I20260927 03:23:05.379793 2767 raft_consensus.cc:2229] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:05.379834 2767 raft_consensus.cc:2258] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:05.380053 2769 tablet_replica.cc:330] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665: stopping tablet replica
I20260927 03:23:05.380101 2767 ts_tablet_manager.cc:1905] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:05.380115 2769 raft_consensus.cc:2229] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:23:05.380191 2769 raft_consensus.cc:2258] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:05.380407 2769 ts_tablet_manager.cc:1905] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:05.381810 2767 ts_tablet_manager.cc:1918] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:23:05.381857 2769 ts_tablet_manager.cc:1918] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:23:05.381878 2767 log.cc:1196] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/1ba5c8ea58c242c7b9c893a826e7ab70
I20260927 03:23:05.381906 2769 log.cc:1196] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/1ba5c8ea58c242c7b9c893a826e7ab70
I20260927 03:23:05.382083 2767 ts_tablet_manager.cc:1939] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 39b23280a2a94ff6b7c3be2b8436b2c9: Deleting consensus metadata
I20260927 03:23:05.382124 2769 ts_tablet_manager.cc:1939] T 1ba5c8ea58c242c7b9c893a826e7ab70 P 7b11085737f74876ab2499caefa8e665: Deleting consensus metadata
I20260927 03:23:05.382426 2106 catalog_manager.cc:4841] TS 7b11085737f74876ab2499caefa8e665 (127.0.98.2:36457): tablet 1ba5c8ea58c242c7b9c893a826e7ab70 (table test-table [id=9a4ebff194af4e348ade30a336e5b4d1]) successfully deleted
I20260927 03:23:05.382427 2105 catalog_manager.cc:4841] TS 39b23280a2a94ff6b7c3be2b8436b2c9 (127.0.98.3:44631): tablet 1ba5c8ea58c242c7b9c893a826e7ab70 (table test-table [id=9a4ebff194af4e348ade30a336e5b4d1]) successfully deleted
W20260927 03:23:05.385368 2104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43593 (error 108)
I20260927 03:23:05.385522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 2332
I20260927 03:23:05.386409 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:05.386207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43593 (local address 127.0.0.1:44100)
0927 03:23:05.386326 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:05.386373 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
W20260927 03:23:05.386516 2104 catalog_manager.cc:4584] TS 6cccd6746a5548a3af3923170e8e0938 (127.0.98.1:43593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ba5c8ea58c242c7b9c893a826e7ab70: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43593: connect: Connection refused (error 111)
I20260927 03:23:05.390904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 2472
I20260927 03:23:05.395376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 2088
2026-09-27T03:23:05Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5860 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:23:05 dist-test-slave-gw2v krb5kdc[2774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:23:05 dist-test-slave-gw2v krb5kdc[2774](info): set up 2 sockets
Sep 27 03:23:05 dist-test-slave-gw2v krb5kdc[2774](info): commencing operation
krb5kdc: starting...
W20260927 03:23:06.362437 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:23:06 dist-test-slave-gw2v krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:23:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:23:06.479617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33835
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:06.558610 2787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:06.558866 2787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:06.558900 2787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:06.560123 2787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:06.560174 2787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:06.560200 2787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:06.560216 2787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:06.560231 2787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:06.561805 2787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:06.562059 2787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:06.562283 2787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:06.566869 2795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:06.566869 2792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:06.566954 2793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:06.567201 2787 server_base.cc:1034] running on GCE node
I20260927 03:23:06.567474 2787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:06.567763 2787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:06.568933 2787 hybrid_clock.cc:648] HybridClock initialized: now 1790479386568912 us; error 30 us; skew 500 ppm
Sep 27 03:23:06 dist-test-slave-gw2v krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:06.575311 2787 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:23:06.575771 2787 webserver.cc:460] Webserver started at http://127.0.98.62:44287/ using document root <none> and password file <none>
I20260927 03:23:06.575985 2787 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:06.576033 2787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:06.576177 2787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:06.577467 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "4c6b314599c643ae879edbd798953770"
format_stamp: "Formatted at 2026-09-27 03:23:06 on dist-test-slave-gw2v"
server_key: "fcea5d2f2e98f0c2ade315f648c5008f"
server_key_iv: "40ecc2fff0f4990190e9d708303a28a2"
server_key_version: "encryptionkey@0"
I20260927 03:23:06.577801 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c6b314599c643ae879edbd798953770"
format_stamp: "Formatted at 2026-09-27 03:23:06 on dist-test-slave-gw2v"
server_key: "fcea5d2f2e98f0c2ade315f648c5008f"
server_key_iv: "40ecc2fff0f4990190e9d708303a28a2"
server_key_version: "encryptionkey@0"
I20260927 03:23:06.579162 2787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:06.579900 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:06.580053 2787 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:06.580138 2787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "4c6b314599c643ae879edbd798953770"
format_stamp: "Formatted at 2026-09-27 03:23:06 on dist-test-slave-gw2v"
server_key: "fcea5d2f2e98f0c2ade315f648c5008f"
server_key_iv: "40ecc2fff0f4990190e9d708303a28a2"
server_key_version: "encryptionkey@0"
I20260927 03:23:06.580202 2787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:06.593770 2787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:06.595832 2787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:06.595979 2787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:06.599840 2787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33835
I20260927 03:23:06.599874 2854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33835 every 8 connection(s)
I20260927 03:23:06.600358 2787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:06.600970 2855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:06.603204 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770: Bootstrap starting.
I20260927 03:23:06.603762 2855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:06.604014 2855 log.cc:824] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:06.604323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 2787
I20260927 03:23:06.604424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:06.604648 392 external_mini_cluster.cc:1428] Setting key d6c0770504b2dae887c93fdc62ef2aa5
I20260927 03:23:06.604782 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770: No bootstrap required, opened a new log
I20260927 03:23:06.607360 2855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6b314599c643ae879edbd798953770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33835 } }
I20260927 03:23:06.607494 2855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:06.607515 2855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6b314599c643ae879edbd798953770, State: Initialized, Role: FOLLOWER
I20260927 03:23:06.607623 2855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6b314599c643ae879edbd798953770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33835 } }
I20260927 03:23:06.607690 2855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:06.607713 2855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:06.607756 2855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:06.608258 2855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6b314599c643ae879edbd798953770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33835 } }
I20260927 03:23:06.608353 2855 leader_election.cc:302] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c6b314599c643ae879edbd798953770; no voters:
I20260927 03:23:06.608492 2855 leader_election.cc:288] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:06.608541 2860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:06.608721 2860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [term 1 LEADER]: Becoming Leader. State: Replica: 4c6b314599c643ae879edbd798953770, State: Running, Role: LEADER
I20260927 03:23:06.608726 2855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:06.608834 2860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6b314599c643ae879edbd798953770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33835 } }
I20260927 03:23:06.610261 2862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c6b314599c643ae879edbd798953770. Latest consensus state: current_term: 1 leader_uuid: "4c6b314599c643ae879edbd798953770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6b314599c643ae879edbd798953770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33835 } } }
I20260927 03:23:06.610435 2861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c6b314599c643ae879edbd798953770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c6b314599c643ae879edbd798953770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33835 } } }
I20260927 03:23:06.610572 2862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [sys.catalog]: This master's current role is: LEADER
W20260927 03:23:06.610579 2875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 03:23:06.610594 2861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770 [sys.catalog]: This master's current role is: LEADER
W20260927 03:23:06.610659 2875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:23:06.610922 2877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:06.611081 2877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:23:06 dist-test-slave-gw2v krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:06.615428 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:06.605292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.0.98.62:33835)
0927 03:23:06.605454 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:06.605461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:06.605484 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:23:06.606098 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:06.606107 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:06.606165 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:06.606298 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:06.606926 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:06.607920 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:06.608818 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:06.609427 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:06.609536 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:06.612205 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:06.612209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:06.612209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:06.612222 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:06.613724 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:06.614028 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:06.614031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:06.614032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:06.614120 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:06.614320 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:06.614320 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:06.614321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:06.614564 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:06.615125 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:06.615182 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 03:23:06.615284 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260927 03:23:06.619448 2877 catalog_manager.cc:1347] Generated new cluster ID: 696e2f610e124514b701ce6b0926cf0c
I20260927 03:23:06.619508 2877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:23:06.648186 2877 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:06.648962 2877 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:06.658304 2877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c6b314599c643ae879edbd798953770: Generated new TSK 0
I20260927 03:23:06.658532 2877 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:23:06.691033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33835
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:06.770447 2881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:06.770660 2881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:06.770690 2881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:06.771857 2881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:06.771901 2881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:06.771934 2881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:23:06.773391 2881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:06.773629 2881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:06.773837 2881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:06.778702 2887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:06.778739 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:06.778777 2881 server_base.cc:1034] running on GCE node
W20260927 03:23:06.778733 2886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:06.779229 2881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:06.779479 2881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:06.780629 2881 hybrid_clock.cc:648] HybridClock initialized: now 1790479386780601 us; error 46 us; skew 500 ppm
Sep 27 03:23:06 dist-test-slave-gw2v krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:06.783195 2881 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:23:06.783643 2881 webserver.cc:460] Webserver started at http://127.0.98.1:45403/ using document root <none> and password file <none>
I20260927 03:23:06.783838 2881 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:06.783885 2881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:06.784009 2881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:06.785043 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "d786039a0752442591cf831c14f5b812"
format_stamp: "Formatted at 2026-09-27 03:23:06 on dist-test-slave-gw2v"
server_key: "03772c5f3d45e5d2010d79c8f434405a"
server_key_iv: "45a6cfaf02402de57493d980953c9a35"
server_key_version: "encryptionkey@0"
I20260927 03:23:06.785395 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d786039a0752442591cf831c14f5b812"
format_stamp: "Formatted at 2026-09-27 03:23:06 on dist-test-slave-gw2v"
server_key: "03772c5f3d45e5d2010d79c8f434405a"
server_key_iv: "45a6cfaf02402de57493d980953c9a35"
server_key_version: "encryptionkey@0"
I20260927 03:23:06.786669 2881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:06.787498 2896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:06.787645 2881 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:06.787726 2881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "d786039a0752442591cf831c14f5b812"
format_stamp: "Formatted at 2026-09-27 03:23:06 on dist-test-slave-gw2v"
server_key: "03772c5f3d45e5d2010d79c8f434405a"
server_key_iv: "45a6cfaf02402de57493d980953c9a35"
server_key_version: "encryptionkey@0"
I20260927 03:23:06.787792 2881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:06.799757 2881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:06.801891 2881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:06.802033 2881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:06.802260 2881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:06.802544 2881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:06.802577 2881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:06.802606 2881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:06.802624 2881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:06.808010 2881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40535
I20260927 03:23:06.808092 3009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40535 every 8 connection(s)
I20260927 03:23:06.808367 2881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:23:06 dist-test-slave-gw2v krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:06.816255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 2881
I20260927 03:23:06.816344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:23:06.816541 392 external_mini_cluster.cc:1428] Setting key 295d0675176fcff82b2753e2de1e6a70
I20260927 03:23:06.817780 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:06.809091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36559 (local address 127.0.98.62:33835)
0927 03:23:06.809180 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:06.809182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:06.809368 (+ 186us) server_negotiation.cc:407] Connection header received
0927 03:23:06.810337 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:06.810338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:06.810353 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:06.810411 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:06.811119 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:06.811657 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:06.812672 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:06.812881 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:06.812969 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:06.815021 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:06.815022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:06.815022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:06.815030 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:06.816255 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:06.816728 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:06.816729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:06.816729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:06.816808 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:06.817105 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:06.817106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:06.817106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:06.817245 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:06.817312 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:06.817638 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 03:23:06.817712 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 03:23:06.818149 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:06.809036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33835 (local address 127.0.98.1:36559)
0927 03:23:06.809203 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:06.809262 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:06.810218 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:06.810480 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:06.810483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:06.810736 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:06.810970 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:06.810975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:06.811775 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:06.811776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:06.812579 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:06.812581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:06.812992 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:06.812993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:06.813113 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:06.813568 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:06.813575 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:06.814944 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:06.816351 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:06.816359 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:06.816361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:06.816661 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:06.816900 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:06.816901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:06.816902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:06.817041 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:06.817318 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:06.817325 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:06.817496 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 03:23:06.817627 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 03:23:06.817814 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260927 03:23:06.818893 3010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33835
I20260927 03:23:06.819001 3010 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:06.819186 3010 heartbeater.cc:511] Master 127.0.98.62:33835 requested a full tablet report, sending...
I20260927 03:23:06.819700 2819 ts_manager.cc:194] Registered new tserver with Master: d786039a0752442591cf831c14f5b812 (127.0.98.1:40535)
I20260927 03:23:06.820340 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36559
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:23:06.842090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33835
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:06.922112 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:06.922334 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:06.922358 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:06.923566 3015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:06.923616 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:06.923643 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:23:06.925161 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:06.925397 3015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:06.925608 3015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:23:06.930680 3015 server_base.cc:1034] running on GCE node
W20260927 03:23:06.930665 3023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:06.930665 3021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:06.930692 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:06.931068 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:06.931295 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:06.932444 3015 hybrid_clock.cc:648] HybridClock initialized: now 1790479386932425 us; error 30 us; skew 500 ppm
Sep 27 03:23:06 dist-test-slave-gw2v krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:06.934926 3015 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:23:06.935392 3015 webserver.cc:460] Webserver started at http://127.0.98.2:35785/ using document root <none> and password file <none>
I20260927 03:23:06.935585 3015 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:06.935642 3015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:06.935763 3015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:06.936801 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "36f2b326210b4f86bcf604c9d69de48c"
format_stamp: "Formatted at 2026-09-27 03:23:06 on dist-test-slave-gw2v"
server_key: "b65adafd10ea8397c87641088e9032db"
server_key_iv: "e6e10f838693cbfe0fd18561b1baf8f9"
server_key_version: "encryptionkey@0"
I20260927 03:23:06.937126 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "36f2b326210b4f86bcf604c9d69de48c"
format_stamp: "Formatted at 2026-09-27 03:23:06 on dist-test-slave-gw2v"
server_key: "b65adafd10ea8397c87641088e9032db"
server_key_iv: "e6e10f838693cbfe0fd18561b1baf8f9"
server_key_version: "encryptionkey@0"
I20260927 03:23:06.938408 3015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:23:06.939193 3030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:06.939354 3015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:06.939446 3015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "36f2b326210b4f86bcf604c9d69de48c"
format_stamp: "Formatted at 2026-09-27 03:23:06 on dist-test-slave-gw2v"
server_key: "b65adafd10ea8397c87641088e9032db"
server_key_iv: "e6e10f838693cbfe0fd18561b1baf8f9"
server_key_version: "encryptionkey@0"
I20260927 03:23:06.939507 3015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:06.961278 3015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:06.963485 3015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:06.963649 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:06.963871 3015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:06.964169 3015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:06.964201 3015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:06.964229 3015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:06.964247 3015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:06.969640 3015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45535
I20260927 03:23:06.969694 3143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45535 every 8 connection(s)
I20260927 03:23:06.969992 3015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:23:06 dist-test-slave-gw2v krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:06.976881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 3015
I20260927 03:23:06.976960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:23:06.977128 392 external_mini_cluster.cc:1428] Setting key 9c70f0d73ac0a9bde25c6b22a4ba18f1
I20260927 03:23:06.979630 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:06.970783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45765 (local address 127.0.98.62:33835)
0927 03:23:06.970860 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:06.970862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:06.971051 (+ 189us) server_negotiation.cc:407] Connection header received
0927 03:23:06.972059 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:06.972060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:06.972073 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:06.972151 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:06.972760 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:06.973246 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:06.974229 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:06.974414 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:06.974475 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:06.976820 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:06.976821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:06.976822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:06.976830 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:06.978157 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:06.978519 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:06.978521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:06.978521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:06.978587 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:06.978886 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:06.978887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:06.978887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:06.979015 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:06.979058 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:06.979461 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 03:23:06.979540 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 03:23:06.979813 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:06.970749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33835 (local address 127.0.98.2:45765)
0927 03:23:06.970933 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:06.970971 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:06.971954 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:06.972210 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:06.972215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:06.972483 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:06.972671 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:06.972676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:06.973333 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:06.973334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:06.974150 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:06.974152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:06.974487 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:06.974488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:06.974612 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:06.975295 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:06.975306 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:06.976703 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:06.978231 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:06.978237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:06.978239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:06.978456 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:06.978657 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:06.978657 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:06.978657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:06.978783 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:06.979105 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:06.979112 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:06.979289 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 03:23:06.979420 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 03:23:06.979591 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
I20260927 03:23:06.980665 3144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33835
I20260927 03:23:06.980767 3144 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:06.980959 3144 heartbeater.cc:511] Master 127.0.98.62:33835 requested a full tablet report, sending...
I20260927 03:23:06.981653 2819 ts_manager.cc:194] Registered new tserver with Master: 36f2b326210b4f86bcf604c9d69de48c (127.0.98.2:45535)
I20260927 03:23:06.982137 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45765
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:23:07.002841 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33835
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:07.083290 3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:07.083503 3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:07.083527 3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:07.084709 3149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:07.084756 3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:07.084784 3149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:23:07.086246 3149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:07.086426 3149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:07.086613 3149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:23:07.091578 3149 server_base.cc:1034] running on GCE node
W20260927 03:23:07.091753 3155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:07.091578 3157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:07.091579 3154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:07.092154 3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:07.092413 3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:07.093577 3149 hybrid_clock.cc:648] HybridClock initialized: now 1790479387093537 us; error 45 us; skew 500 ppm
Sep 27 03:23:07 dist-test-slave-gw2v krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:07.096253 3149 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:23:07.096688 3149 webserver.cc:460] Webserver started at http://127.0.98.3:33513/ using document root <none> and password file <none>
I20260927 03:23:07.096896 3149 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:07.096940 3149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:07.097054 3149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:07.098107 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "b871ece1d89d4086aac327a7655fe2c7"
format_stamp: "Formatted at 2026-09-27 03:23:07 on dist-test-slave-gw2v"
server_key: "4cf4e737b3f63586b374b994c488f3c0"
server_key_iv: "42b82a74f57d3df0f83304a6c38e79ab"
server_key_version: "encryptionkey@0"
I20260927 03:23:07.098423 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b871ece1d89d4086aac327a7655fe2c7"
format_stamp: "Formatted at 2026-09-27 03:23:07 on dist-test-slave-gw2v"
server_key: "4cf4e737b3f63586b374b994c488f3c0"
server_key_iv: "42b82a74f57d3df0f83304a6c38e79ab"
server_key_version: "encryptionkey@0"
I20260927 03:23:07.099701 3149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:23:07.100584 3164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:07.100749 3149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:07.100840 3149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "b871ece1d89d4086aac327a7655fe2c7"
format_stamp: "Formatted at 2026-09-27 03:23:07 on dist-test-slave-gw2v"
server_key: "4cf4e737b3f63586b374b994c488f3c0"
server_key_iv: "42b82a74f57d3df0f83304a6c38e79ab"
server_key_version: "encryptionkey@0"
I20260927 03:23:07.100911 3149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:07.120288 3149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:07.122124 3149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:07.122244 3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:07.122469 3149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:07.122771 3149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:07.122804 3149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:07.122857 3149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:07.122879 3149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:07.128203 3149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45565
I20260927 03:23:07.128300 3277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45565 every 8 connection(s)
I20260927 03:23:07.128546 3149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:23:07 dist-test-slave-gw2v krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:07.137426 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:07.129217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32865 (local address 127.0.98.62:33835)
0927 03:23:07.129259 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:07.129261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:07.129463 (+ 202us) server_negotiation.cc:407] Connection header received
0927 03:23:07.130397 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:07.130398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:07.130409 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:07.130476 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:07.131115 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:07.131602 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:07.132603 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:07.132789 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:07.132843 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:07.134719 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:07.134721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:07.134721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:07.134728 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:07.136031 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:07.136469 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:07.136470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:07.136470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:07.136529 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:07.136826 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:07.136827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:07.136827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:07.136949 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:07.137015 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:07.137289 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 03:23:07.137356 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:23:07.137538 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:07.129184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33835 (local address 127.0.98.3:32865)
0927 03:23:07.129346 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:07.129388 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:07.130262 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:07.130534 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:07.130537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:07.130789 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:07.131031 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:07.131036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:07.131673 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:07.131674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:07.132518 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:07.132520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:07.132850 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:07.132851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:07.132940 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:07.133357 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:07.133365 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:07.134617 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:07.136144 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:07.136151 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:07.136152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:07.136387 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:07.136618 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:07.136618 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:07.136619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:07.136730 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:07.137010 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:07.137013 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:07.137138 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 03:23:07.137243 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 03:23:07.137367 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260927 03:23:07.137753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 3149
I20260927 03:23:07.137836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:23:07.137948 3278 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33835
I20260927 03:23:07.138011 392 external_mini_cluster.cc:1428] Setting key 66decd1d99dc1fac995e93beeea2d9ea
I20260927 03:23:07.138043 3278 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:07.138230 3278 heartbeater.cc:511] Master 127.0.98.62:33835 requested a full tablet report, sending...
I20260927 03:23:07.138619 2818 ts_manager.cc:194] Registered new tserver with Master: b871ece1d89d4086aac327a7655fe2c7 (127.0.98.3:45565)
I20260927 03:23:07.139029 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32865
I20260927 03:23:07.139384 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 03:23:07.150228 2859 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:07.149353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.0.98.62:33835)
0927 03:23:07.149402 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:07.149404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:07.149493 (+ 89us) server_negotiation.cc:407] Connection header received
0927 03:23:07.149563 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:07.149564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:07.149575 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:07.149630 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:07.150154 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53852: BlockingRecv error: recv got EOF from 127.0.0.1:53852 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:23:07.151155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 2881
I20260927 03:23:07.155579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 3015
I20260927 03:23:07.159796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 3149
I20260927 03:23:07.163705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 2787
2026-09-27T03:23:07Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1768 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:23:07 dist-test-slave-gw2v krb5kdc[3290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:23:07 dist-test-slave-gw2v krb5kdc[3290](info): set up 2 sockets
Sep 27 03:23:07 dist-test-slave-gw2v krb5kdc[3290](info): commencing operation
krb5kdc: starting...
W20260927 03:23:08.106573 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:23:08 dist-test-slave-gw2v krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:23:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:23:08.217196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41287
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:08.297758 3303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:08.297986 3303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:08.298017 3303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:08.299213 3303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:08.299257 3303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:08.299275 3303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:08.299291 3303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:08.299305 3303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:08.300791 3303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:08.301038 3303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:08.301247 3303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:08.305884 3311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:08.305922 3303 server_base.cc:1034] running on GCE node
W20260927 03:23:08.305892 3309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:08.305891 3308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:08.306473 3303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:08.306746 3303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:08.307910 3303 hybrid_clock.cc:648] HybridClock initialized: now 1790479388307907 us; error 41 us; skew 500 ppm
Sep 27 03:23:08 dist-test-slave-gw2v krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:08.310391 3303 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:23:08.310910 3303 webserver.cc:460] Webserver started at http://127.0.98.62:46067/ using document root <none> and password file <none>
I20260927 03:23:08.311123 3303 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:08.311172 3303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:08.311299 3303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:08.312338 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "89fa96956b1c46b2bc2cdac2e3dec076"
format_stamp: "Formatted at 2026-09-27 03:23:08 on dist-test-slave-gw2v"
server_key: "122e8c935a7cceeda54fb5899a054dc8"
server_key_iv: "d48aea22772982d7c3a21d98712a603f"
server_key_version: "encryptionkey@0"
I20260927 03:23:08.312670 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "89fa96956b1c46b2bc2cdac2e3dec076"
format_stamp: "Formatted at 2026-09-27 03:23:08 on dist-test-slave-gw2v"
server_key: "122e8c935a7cceeda54fb5899a054dc8"
server_key_iv: "d48aea22772982d7c3a21d98712a603f"
server_key_version: "encryptionkey@0"
I20260927 03:23:08.313905 3303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:08.314613 3318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:08.314792 3303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:08.314913 3303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "89fa96956b1c46b2bc2cdac2e3dec076"
format_stamp: "Formatted at 2026-09-27 03:23:08 on dist-test-slave-gw2v"
server_key: "122e8c935a7cceeda54fb5899a054dc8"
server_key_iv: "d48aea22772982d7c3a21d98712a603f"
server_key_version: "encryptionkey@0"
I20260927 03:23:08.314983 3303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:08.324654 3303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:08.326538 3303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:08.326673 3303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:08.330376 3303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41287
I20260927 03:23:08.330410 3370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41287 every 8 connection(s)
I20260927 03:23:08.330734 3303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:08.331609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 3303
I20260927 03:23:08.331645 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:08.331709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:08.331888 392 external_mini_cluster.cc:1428] Setting key 3804a6b97056e4c78f659fa3b02f67e2
I20260927 03:23:08.334988 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076: Bootstrap starting.
I20260927 03:23:08.335683 3371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:08.335973 3371 log.cc:824] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:08.336696 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076: No bootstrap required, opened a new log
Sep 27 03:23:08 dist-test-slave-gw2v krb5kdc[3290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:08.338692 3371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fa96956b1c46b2bc2cdac2e3dec076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41287 } }
I20260927 03:23:08.338897 3371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:08.338946 3371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89fa96956b1c46b2bc2cdac2e3dec076, State: Initialized, Role: FOLLOWER
I20260927 03:23:08.339074 3371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fa96956b1c46b2bc2cdac2e3dec076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41287 } }
I20260927 03:23:08.339149 3371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:08.339188 3371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:08.339234 3371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:08.339911 3371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fa96956b1c46b2bc2cdac2e3dec076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41287 } }
I20260927 03:23:08.340021 3371 leader_election.cc:302] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89fa96956b1c46b2bc2cdac2e3dec076; no voters:
I20260927 03:23:08.340188 3371 leader_election.cc:288] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:08.340245 3376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:08.340410 3376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [term 1 LEADER]: Becoming Leader. State: Replica: 89fa96956b1c46b2bc2cdac2e3dec076, State: Running, Role: LEADER
I20260927 03:23:08.340427 3371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:08.340520 3376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fa96956b1c46b2bc2cdac2e3dec076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41287 } }
I20260927 03:23:08.340826 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89fa96956b1c46b2bc2cdac2e3dec076. Latest consensus state: current_term: 1 leader_uuid: "89fa96956b1c46b2bc2cdac2e3dec076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fa96956b1c46b2bc2cdac2e3dec076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41287 } } }
I20260927 03:23:08.341051 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:08.340989 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89fa96956b1c46b2bc2cdac2e3dec076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fa96956b1c46b2bc2cdac2e3dec076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41287 } } }
I20260927 03:23:08.341426 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076 [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:08.341681 3385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:08.342303 3385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:08.343366 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:08.332451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.0.98.62:41287)
0927 03:23:08.332627 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:08.332639 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:08.332666 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:23:08.333117 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:08.333124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:08.333173 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:08.333286 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:08.334139 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:08.335257 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:08.336263 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:08.336691 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:08.336791 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:08.339354 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:08.339358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:08.339359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:08.339371 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:08.341347 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:08.341785 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:08.341788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:08.341789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:08.341893 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:08.342070 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:08.342070 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:08.342071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:08.342368 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:08.342508 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:08.343032 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 03:23:08.343177 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260927 03:23:08.343735 3385 catalog_manager.cc:1347] Generated new cluster ID: fde771e7057848c09f3c98ec6b5b497d
I20260927 03:23:08.343779 3385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:23:08.344265 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:23:08.355726 3385 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:08.356460 3385 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:08.357267 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:08.353851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.0.98.62:41287)
0927 03:23:08.353897 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:08.353899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:08.353996 (+ 97us) server_negotiation.cc:407] Connection header received
0927 03:23:08.354057 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:08.354058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:08.354068 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:08.354114 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:08.354469 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:08.354986 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:08.355867 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:08.356099 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:08.356131 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:08.356814 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:08.356816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:08.356816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:08.356998 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:08.357013 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:08.357020 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:08.357066 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:08.357130 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 03:23:08.357172 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":23}
I20260927 03:23:08.360095 3385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89fa96956b1c46b2bc2cdac2e3dec076: Generated new TSK 0
I20260927 03:23:08.360271 3385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:23:08.364225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 3303
2026-09-27T03:23:08Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1198 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:23:08 dist-test-slave-gw2v krb5kdc[3404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:23:08 dist-test-slave-gw2v krb5kdc[3404](info): set up 2 sockets
Sep 27 03:23:08 dist-test-slave-gw2v krb5kdc[3404](info): commencing operation
krb5kdc: starting...
W20260927 03:23:09.310483 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:23:09 dist-test-slave-gw2v krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:23:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:23:09.429241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41421
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:09.507449 3417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:09.507660 3417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:09.507689 3417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:09.508817 3417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:09.508857 3417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:09.508876 3417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:09.508893 3417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:09.508908 3417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:09.510335 3417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:09.510571 3417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:09.510767 3417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:09.515347 3423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:09.515355 3422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:09.515396 3425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:09.515544 3417 server_base.cc:1034] running on GCE node
I20260927 03:23:09.515753 3417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:09.515991 3417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:09.517136 3417 hybrid_clock.cc:648] HybridClock initialized: now 1790479389517125 us; error 29 us; skew 500 ppm
Sep 27 03:23:09 dist-test-slave-gw2v krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:09.519521 3417 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:23:09.519927 3417 webserver.cc:460] Webserver started at http://127.0.98.62:46733/ using document root <none> and password file <none>
I20260927 03:23:09.520113 3417 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:09.520159 3417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:09.520279 3417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:09.521296 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "38981785a652426c83bfa5d29ac3b8e7"
format_stamp: "Formatted at 2026-09-27 03:23:09 on dist-test-slave-gw2v"
server_key: "e83ae11c1af928688e0375a0fdf66aca"
server_key_iv: "553dcabe0eaa425284b3040b84690908"
server_key_version: "encryptionkey@0"
I20260927 03:23:09.521612 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "38981785a652426c83bfa5d29ac3b8e7"
format_stamp: "Formatted at 2026-09-27 03:23:09 on dist-test-slave-gw2v"
server_key: "e83ae11c1af928688e0375a0fdf66aca"
server_key_iv: "553dcabe0eaa425284b3040b84690908"
server_key_version: "encryptionkey@0"
I20260927 03:23:09.522786 3417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:23:09.523617 3432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:09.523798 3417 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:09.523886 3417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "38981785a652426c83bfa5d29ac3b8e7"
format_stamp: "Formatted at 2026-09-27 03:23:09 on dist-test-slave-gw2v"
server_key: "e83ae11c1af928688e0375a0fdf66aca"
server_key_iv: "553dcabe0eaa425284b3040b84690908"
server_key_version: "encryptionkey@0"
I20260927 03:23:09.523954 3417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:09.544978 3417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:09.547060 3417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:09.547187 3417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:09.550812 3417 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41421
I20260927 03:23:09.550870 3484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41421 every 8 connection(s)
I20260927 03:23:09.551215 3417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:09.552138 3485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:09.553969 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 3417
I20260927 03:23:09.554068 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:09.554231 392 external_mini_cluster.cc:1428] Setting key c210cb3630d30242a4295f8ad7dc40e0
I20260927 03:23:09.554452 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7: Bootstrap starting.
I20260927 03:23:09.555254 3485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:09.555562 3485 log.cc:824] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:09.556318 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7: No bootstrap required, opened a new log
I20260927 03:23:09.558022 3485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38981785a652426c83bfa5d29ac3b8e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260927 03:23:09.558161 3485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:09.558192 3485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38981785a652426c83bfa5d29ac3b8e7, State: Initialized, Role: FOLLOWER
I20260927 03:23:09.558320 3485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38981785a652426c83bfa5d29ac3b8e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260927 03:23:09.558409 3485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:09.558441 3485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:09.558490 3485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:09.559106 3485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38981785a652426c83bfa5d29ac3b8e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260927 03:23:09.559214 3485 leader_election.cc:302] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38981785a652426c83bfa5d29ac3b8e7; no voters:
I20260927 03:23:09.559372 3485 leader_election.cc:288] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:09.559393 3490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:09.559516 3490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [term 1 LEADER]: Becoming Leader. State: Replica: 38981785a652426c83bfa5d29ac3b8e7, State: Running, Role: LEADER
I20260927 03:23:09.559630 3485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:09.559661 3490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38981785a652426c83bfa5d29ac3b8e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260927 03:23:09.560077 3490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38981785a652426c83bfa5d29ac3b8e7. Latest consensus state: current_term: 1 leader_uuid: "38981785a652426c83bfa5d29ac3b8e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38981785a652426c83bfa5d29ac3b8e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } }
I20260927 03:23:09.560156 3490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:09.560312 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38981785a652426c83bfa5d29ac3b8e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38981785a652426c83bfa5d29ac3b8e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } }
I20260927 03:23:09.560385 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7 [sys.catalog]: This master's current role is: LEADER
Sep 27 03:23:09 dist-test-slave-gw2v krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:09.561053 3499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:09.562150 3499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:09.563668 3499 catalog_manager.cc:1347] Generated new cluster ID: 140292551a3c406081bb73452d9c750e
I20260927 03:23:09.563720 3499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:23:09.564898 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:09.554756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.0.98.62:41421)
0927 03:23:09.555037 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:09.555046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:09.555080 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:23:09.555642 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:09.555651 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:09.555717 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:09.555849 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:09.556508 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:09.557392 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:09.558101 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:09.558581 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:09.558676 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:09.561358 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:09.561362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:09.561362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:09.561374 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:09.563288 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:09.563693 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:09.563697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:09.563697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:09.563808 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:09.564017 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:09.564018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:09.564018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:09.564307 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:09.564443 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:09.564549 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 03:23:09.564679 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":181,"threads_started":1}
I20260927 03:23:09.590889 3499 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:09.591490 3499 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:09.597638 3499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38981785a652426c83bfa5d29ac3b8e7: Generated new TSK 0
I20260927 03:23:09.597829 3499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:23:09.629956 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:09.712734 3511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:09.712935 3511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:09.712958 3511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:09.714104 3511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:09.714143 3511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:09.714167 3511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:23:09.715581 3511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:09.715765 3511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:09.715947 3511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:09.720971 3516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:09.720942 3517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:09.721016 3511 server_base.cc:1034] running on GCE node
W20260927 03:23:09.720961 3519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:09.721323 3511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:09.721551 3511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:09.722710 3511 hybrid_clock.cc:648] HybridClock initialized: now 1790479389722694 us; error 27 us; skew 500 ppm
Sep 27 03:23:09 dist-test-slave-gw2v krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:09.725155 3511 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:23:09.725556 3511 webserver.cc:460] Webserver started at http://127.0.98.1:34875/ using document root <none> and password file <none>
I20260927 03:23:09.725745 3511 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:09.725869 3511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:09.726008 3511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:09.727128 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "36b5caff75d44301bb8349dd1bc16230"
format_stamp: "Formatted at 2026-09-27 03:23:09 on dist-test-slave-gw2v"
server_key: "5bac6d166f07fc529231ddb9c5fe328b"
server_key_iv: "b960d9c4647550a117ca4e79f3f52ea9"
server_key_version: "encryptionkey@0"
I20260927 03:23:09.727443 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "36b5caff75d44301bb8349dd1bc16230"
format_stamp: "Formatted at 2026-09-27 03:23:09 on dist-test-slave-gw2v"
server_key: "5bac6d166f07fc529231ddb9c5fe328b"
server_key_iv: "b960d9c4647550a117ca4e79f3f52ea9"
server_key_version: "encryptionkey@0"
I20260927 03:23:09.728619 3511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:23:09.729302 3526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:09.729514 3511 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:09.729610 3511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "36b5caff75d44301bb8349dd1bc16230"
format_stamp: "Formatted at 2026-09-27 03:23:09 on dist-test-slave-gw2v"
server_key: "5bac6d166f07fc529231ddb9c5fe328b"
server_key_iv: "b960d9c4647550a117ca4e79f3f52ea9"
server_key_version: "encryptionkey@0"
I20260927 03:23:09.729682 3511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:09.746503 3511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:09.748442 3511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:09.748603 3511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:09.748906 3511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:09.749215 3511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:09.749253 3511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:09.749287 3511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:09.749306 3511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:09.755458 3511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42091
I20260927 03:23:09.755513 3639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42091 every 8 connection(s)
I20260927 03:23:09.755841 3511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:23:09 dist-test-slave-gw2v krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:09.764708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 3511
I20260927 03:23:09.764797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:23:09.764851 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:09.756546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49787 (local address 127.0.98.62:41421)
0927 03:23:09.756607 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:09.756609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:09.756803 (+ 194us) server_negotiation.cc:407] Connection header received
0927 03:23:09.757321 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:09.757323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:09.757334 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:09.757385 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:09.758315 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:09.758852 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:09.759858 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:09.760113 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:09.760170 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:09.762341 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:09.762342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:09.762343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:09.762351 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:09.763580 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:09.763928 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:09.763929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:09.763929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:09.764020 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:09.764233 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:09.764234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:09.764234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:09.764343 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:09.764398 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:09.764671 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 03:23:09.764760 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:23:09.765009 392 external_mini_cluster.cc:1428] Setting key 7186473c452dd678b81bf793efd418a1
I20260927 03:23:09.765014 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:09.756533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.1:49787)
0927 03:23:09.756684 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:09.756722 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:09.757220 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:09.757463 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:09.757467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:09.758019 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:09.758218 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:09.758223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:09.758932 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:09.758933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:09.759775 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:09.759777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:09.760183 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:09.760183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:09.760340 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:09.760784 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:09.760792 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:09.762256 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:09.763648 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:09.763654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:09.763656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:09.763861 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:09.764076 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:09.764077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:09.764077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:09.764174 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:09.764393 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:09.764398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:09.764556 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 03:23:09.764645 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 03:23:09.764794 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260927 03:23:09.765539 3640 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41421
I20260927 03:23:09.765645 3640 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:09.765822 3640 heartbeater.cc:511] Master 127.0.98.62:41421 requested a full tablet report, sending...
I20260927 03:23:09.766405 3449 ts_manager.cc:194] Registered new tserver with Master: 36b5caff75d44301bb8349dd1bc16230 (127.0.98.1:42091)
I20260927 03:23:09.767279 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49787
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:23:09.790323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:09.869166 3645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:09.869375 3645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:09.869397 3645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:09.870538 3645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:09.870577 3645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:09.870603 3645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:23:09.872025 3645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:09.872249 3645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:09.872463 3645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:09.877314 3653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:09.877305 3650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:09.877529 3651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:09.877429 3645 server_base.cc:1034] running on GCE node
I20260927 03:23:09.877755 3645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:09.877988 3645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:09.879135 3645 hybrid_clock.cc:648] HybridClock initialized: now 1790479389879114 us; error 32 us; skew 500 ppm
Sep 27 03:23:09 dist-test-slave-gw2v krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:09.881714 3645 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:23:09.882217 3645 webserver.cc:460] Webserver started at http://127.0.98.2:43989/ using document root <none> and password file <none>
I20260927 03:23:09.882467 3645 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:09.882520 3645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:09.882652 3645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:09.883775 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "46ba6fe7e7034ad8a77e3d3c72d19693"
format_stamp: "Formatted at 2026-09-27 03:23:09 on dist-test-slave-gw2v"
server_key: "5ffcf77f9f65cc8dc0a21d9af2ed93b5"
server_key_iv: "e9f1733e4b592a91ffe4385a92061905"
server_key_version: "encryptionkey@0"
I20260927 03:23:09.884235 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "46ba6fe7e7034ad8a77e3d3c72d19693"
format_stamp: "Formatted at 2026-09-27 03:23:09 on dist-test-slave-gw2v"
server_key: "5ffcf77f9f65cc8dc0a21d9af2ed93b5"
server_key_iv: "e9f1733e4b592a91ffe4385a92061905"
server_key_version: "encryptionkey@0"
I20260927 03:23:09.885788 3645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:09.886595 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:09.886744 3645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:09.886855 3645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "46ba6fe7e7034ad8a77e3d3c72d19693"
format_stamp: "Formatted at 2026-09-27 03:23:09 on dist-test-slave-gw2v"
server_key: "5ffcf77f9f65cc8dc0a21d9af2ed93b5"
server_key_iv: "e9f1733e4b592a91ffe4385a92061905"
server_key_version: "encryptionkey@0"
I20260927 03:23:09.886966 3645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:09.896682 3645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:09.899156 3645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:09.899298 3645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:09.899523 3645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:09.899833 3645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:09.899866 3645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:09.899894 3645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:09.899911 3645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:09.905748 3645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42421
I20260927 03:23:09.905895 3773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42421 every 8 connection(s)
I20260927 03:23:09.906137 3645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:23:09 dist-test-slave-gw2v krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:09.915107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 3645
I20260927 03:23:09.915205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:23:09.915370 392 external_mini_cluster.cc:1428] Setting key 75d6dd55b54fe6a7ea8837b0d8c7b99f
I20260927 03:23:09.915390 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:09.906971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50649 (local address 127.0.98.62:41421)
0927 03:23:09.907027 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:09.907030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:09.907161 (+ 131us) server_negotiation.cc:407] Connection header received
0927 03:23:09.907713 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:09.907714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:09.907725 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:09.907772 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:09.908850 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:09.909362 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:09.910358 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:09.910560 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:09.910615 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:09.912751 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:09.912753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:09.912753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:09.912760 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:09.913990 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:09.914388 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:09.914388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:09.914389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:09.914442 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:09.914683 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:09.914683 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:09.914683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:09.914781 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:09.914846 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:09.915195 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 03:23:09.915294 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:23:09.915565 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:09.906850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.2:50649)
0927 03:23:09.907024 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:09.907073 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:09.907613 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:09.907957 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:09.907961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:09.908552 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:09.908756 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:09.908761 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:09.909446 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:09.909448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:09.910274 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:09.910276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:09.910628 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:09.910629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:09.910753 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:09.911274 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:09.911284 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:09.912655 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:09.914075 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:09.914082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:09.914084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:09.914324 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:09.914511 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:09.914512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:09.914512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:09.914626 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:09.914905 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:09.914910 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:09.915058 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 03:23:09.915153 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 03:23:09.915338 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260927 03:23:09.916093 3774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41421
I20260927 03:23:09.916203 3774 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:09.916381 3774 heartbeater.cc:511] Master 127.0.98.62:41421 requested a full tablet report, sending...
I20260927 03:23:09.916770 3449 ts_manager.cc:194] Registered new tserver with Master: 46ba6fe7e7034ad8a77e3d3c72d19693 (127.0.98.2:42421)
I20260927 03:23:09.917140 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50649
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:23:09.939735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:10.020334 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:10.020562 3779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:10.020592 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:10.021754 3779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:10.021799 3779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:10.021833 3779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:23:10.023288 3779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:10.023528 3779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:10.023731 3779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:10.028404 3784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:10.028463 3779 server_base.cc:1034] running on GCE node
W20260927 03:23:10.028404 3785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:10.028422 3787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:10.028735 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:10.028955 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:10.030102 3779 hybrid_clock.cc:648] HybridClock initialized: now 1790479390030099 us; error 31 us; skew 500 ppm
Sep 27 03:23:10 dist-test-slave-gw2v krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:10.032738 3779 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:23:10.033217 3779 webserver.cc:460] Webserver started at http://127.0.98.3:37493/ using document root <none> and password file <none>
I20260927 03:23:10.033437 3779 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:10.033483 3779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:10.033591 3779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:10.034622 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "50994d9667194ce29d7684c6c71ebee5"
format_stamp: "Formatted at 2026-09-27 03:23:10 on dist-test-slave-gw2v"
server_key: "05a1a6836943d4b3abbd0fd20c2086e8"
server_key_iv: "24fa5a61c4129e3d0995ad10a782928d"
server_key_version: "encryptionkey@0"
I20260927 03:23:10.034963 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "50994d9667194ce29d7684c6c71ebee5"
format_stamp: "Formatted at 2026-09-27 03:23:10 on dist-test-slave-gw2v"
server_key: "05a1a6836943d4b3abbd0fd20c2086e8"
server_key_iv: "24fa5a61c4129e3d0995ad10a782928d"
server_key_version: "encryptionkey@0"
I20260927 03:23:10.036212 3779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:10.037010 3794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:10.037191 3779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:10.037273 3779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "50994d9667194ce29d7684c6c71ebee5"
format_stamp: "Formatted at 2026-09-27 03:23:10 on dist-test-slave-gw2v"
server_key: "05a1a6836943d4b3abbd0fd20c2086e8"
server_key_iv: "24fa5a61c4129e3d0995ad10a782928d"
server_key_version: "encryptionkey@0"
I20260927 03:23:10.037374 3779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:10.057446 3779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:10.059724 3779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:10.059839 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:10.060019 3779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:10.060329 3779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:10.060370 3779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:10.060392 3779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:10.060415 3779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:10.066071 3779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34359
I20260927 03:23:10.066128 3907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34359 every 8 connection(s)
I20260927 03:23:10.066425 3779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:23:10 dist-test-slave-gw2v krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:10.074558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 3779
I20260927 03:23:10.074630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:23:10.074770 392 external_mini_cluster.cc:1428] Setting key 2f8b8ca94369fe99819725f8260aacc2
I20260927 03:23:10.075753 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:10.067123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56841 (local address 127.0.98.62:41421)
0927 03:23:10.067182 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:10.067184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:10.067395 (+ 211us) server_negotiation.cc:407] Connection header received
0927 03:23:10.067864 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:10.067865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:10.067874 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:10.067992 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:10.068941 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:10.069431 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:10.070482 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:10.070663 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:10.070711 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:10.072992 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:10.072993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:10.072994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:10.073000 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:10.074316 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:10.074745 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:10.074747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:10.074747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:10.074800 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:10.075076 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:10.075077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:10.075077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:10.075212 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:10.075275 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:10.075580 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 03:23:10.075660 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:23:10.075940 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:10.067105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.3:56841)
0927 03:23:10.067261 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:10.067303 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:10.067786 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:10.068055 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:10.068058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:10.068628 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:10.068842 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:10.068847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:10.069538 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:10.069540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:10.070398 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:10.070401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:10.070861 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:10.070862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:10.070991 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:10.071440 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:10.071448 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:10.072911 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:10.074404 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:10.074416 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:10.074418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:10.074636 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:10.074886 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:10.074887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:10.074887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:10.075019 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:10.075280 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:10.075286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:10.075449 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 03:23:10.075545 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 03:23:10.075704 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260927 03:23:10.076404 3908 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41421
I20260927 03:23:10.076508 3908 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:10.076678 3908 heartbeater.cc:511] Master 127.0.98.62:41421 requested a full tablet report, sending...
I20260927 03:23:10.077096 3449 ts_manager.cc:194] Registered new tserver with Master: 50994d9667194ce29d7684c6c71ebee5 (127.0.98.3:34359)
I20260927 03:23:10.077473 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56841
I20260927 03:23:10.086596 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:23:10 dist-test-slave-gw2v krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479390, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 03:23:10 dist-test-slave-gw2v krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479390, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:10.108094 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:10.101442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.0.98.62:41421)
0927 03:23:10.101491 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:10.101492 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:10.101618 (+ 126us) server_negotiation.cc:407] Connection header received
0927 03:23:10.101690 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:10.101691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:10.101702 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:10.101749 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:10.102337 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:10.102886 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:10.103596 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:10.103779 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:10.103832 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:10.105896 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:10.105898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:10.105898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:10.105905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:10.107094 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:10.107371 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:10.107372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:10.107373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:10.107440 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:10.107598 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:10.107599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:10.107599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:10.107739 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:10.107825 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:10.107925 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 03:23:10.108000 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 03:23:10.108306 3449 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60220
I20260927 03:23:10.108757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 3511
I20260927 03:23:10.112834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 3645
I20260927 03:23:10.116556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 3779
I20260927 03:23:10.120373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 3417
2026-09-27T03:23:10Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1757 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:23:10 dist-test-slave-gw2v krb5kdc[3920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:23:10 dist-test-slave-gw2v krb5kdc[3920](info): set up 2 sockets
Sep 27 03:23:10 dist-test-slave-gw2v krb5kdc[3920](info): commencing operation
krb5kdc: starting...
W20260927 03:23:11.082487 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:23:11 dist-test-slave-gw2v krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:23:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:23:11.195739 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37221
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:11.273307 3933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:11.273521 3933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:11.273552 3933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:11.274724 3933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:11.274765 3933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:11.274783 3933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:11.274798 3933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:11.274814 3933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:11.276285 3933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:11.276563 3933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:11.276760 3933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:11.281623 3941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:11.281615 3938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:11.281764 3933 server_base.cc:1034] running on GCE node
W20260927 03:23:11.281662 3939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:11.282066 3933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:11.282336 3933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:11.283483 3933 hybrid_clock.cc:648] HybridClock initialized: now 1790479391283466 us; error 32 us; skew 500 ppm
Sep 27 03:23:11 dist-test-slave-gw2v krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:11.285938 3933 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:23:11.286386 3933 webserver.cc:460] Webserver started at http://127.0.98.62:44291/ using document root <none> and password file <none>
I20260927 03:23:11.286593 3933 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:11.286643 3933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:11.286765 3933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:11.287809 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "c87810c71fb74ac3a2631e15e830a884"
format_stamp: "Formatted at 2026-09-27 03:23:11 on dist-test-slave-gw2v"
server_key: "361ad7695e34636b028524fe16e7dcfe"
server_key_iv: "ea8fa955e1aeaf10af1763a3e58a4646"
server_key_version: "encryptionkey@0"
I20260927 03:23:11.288127 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "c87810c71fb74ac3a2631e15e830a884"
format_stamp: "Formatted at 2026-09-27 03:23:11 on dist-test-slave-gw2v"
server_key: "361ad7695e34636b028524fe16e7dcfe"
server_key_iv: "ea8fa955e1aeaf10af1763a3e58a4646"
server_key_version: "encryptionkey@0"
I20260927 03:23:11.289316 3933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:23:11.289958 3948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:11.290122 3933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:11.290202 3933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "c87810c71fb74ac3a2631e15e830a884"
format_stamp: "Formatted at 2026-09-27 03:23:11 on dist-test-slave-gw2v"
server_key: "361ad7695e34636b028524fe16e7dcfe"
server_key_iv: "ea8fa955e1aeaf10af1763a3e58a4646"
server_key_version: "encryptionkey@0"
I20260927 03:23:11.290263 3933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:11.313692 3933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:11.315480 3933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:11.315621 3933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:11.319298 3933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37221
I20260927 03:23:11.319341 4000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37221 every 8 connection(s)
I20260927 03:23:11.319666 3933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:11.320007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 3933
I20260927 03:23:11.320106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:11.320310 392 external_mini_cluster.cc:1428] Setting key 1c30fd43741e494128af0ed43ccdf6d4
I20260927 03:23:11.320564 4001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:11.323851 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884: Bootstrap starting.
I20260927 03:23:11.324429 4001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:11.324720 4001 log.cc:824] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:11.325609 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884: No bootstrap required, opened a new log
Sep 27 03:23:11 dist-test-slave-gw2v krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:11.327489 4001 raft_consensus.cc:348] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87810c71fb74ac3a2631e15e830a884" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37221 } }
I20260927 03:23:11.327657 4001 raft_consensus.cc:374] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:11.327701 4001 raft_consensus.cc:729] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87810c71fb74ac3a2631e15e830a884, State: Initialized, Role: FOLLOWER
I20260927 03:23:11.327829 4001 consensus_queue.cc:260] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87810c71fb74ac3a2631e15e830a884" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37221 } }
I20260927 03:23:11.327903 4001 raft_consensus.cc:388] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:11.327934 4001 raft_consensus.cc:482] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:11.327979 4001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:11.328641 4001 raft_consensus.cc:504] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87810c71fb74ac3a2631e15e830a884" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37221 } }
I20260927 03:23:11.328753 4001 leader_election.cc:302] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c87810c71fb74ac3a2631e15e830a884; no voters:
I20260927 03:23:11.328909 4001 leader_election.cc:288] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:11.328955 4006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:11.329128 4006 raft_consensus.cc:686] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [term 1 LEADER]: Becoming Leader. State: Replica: c87810c71fb74ac3a2631e15e830a884, State: Running, Role: LEADER
I20260927 03:23:11.329151 4001 sys_catalog.cc:564] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:11.329269 4006 consensus_queue.cc:237] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87810c71fb74ac3a2631e15e830a884" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37221 } }
I20260927 03:23:11.329628 4006 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c87810c71fb74ac3a2631e15e830a884. Latest consensus state: current_term: 1 leader_uuid: "c87810c71fb74ac3a2631e15e830a884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87810c71fb74ac3a2631e15e830a884" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37221 } } }
I20260927 03:23:11.329650 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c87810c71fb74ac3a2631e15e830a884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87810c71fb74ac3a2631e15e830a884" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37221 } } }
I20260927 03:23:11.329783 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:11.329980 4006 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884 [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:11.330130 4015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:11.330840 4015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:11.331413 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:11.320854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.0.98.62:37221)
0927 03:23:11.320999 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:11.321008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:11.321038 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:23:11.321504 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:11.321514 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:11.321578 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:11.321723 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:11.322343 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.323689 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.324499 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.325054 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.325159 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:11.327359 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:11.327364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:11.327364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:11.327377 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:11.329244 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:11.329640 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:11.329644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:11.329644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:11.329798 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:11.330009 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:11.330010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:11.330010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:11.330336 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:11.330868 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:11.330971 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 03:23:11.331217 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260927 03:23:11.332432 4015 catalog_manager.cc:1347] Generated new cluster ID: ff32cd8396224eaea1e32950b2cddb88
I20260927 03:23:11.332484 4015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:23:11.349781 4015 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:11.350365 4015 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:11.355584 4015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c87810c71fb74ac3a2631e15e830a884: Generated new TSK 0
I20260927 03:23:11.355746 4015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:23:11.381129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37221
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:11.464190 4027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:11.464407 4027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:11.464443 4027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:11.465615 4027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:11.465652 4027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:11.465685 4027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:23:11.467141 4027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:11.467363 4027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:11.467566 4027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:11.472335 4032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:11.472374 4033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:11.472426 4027 server_base.cc:1034] running on GCE node
W20260927 03:23:11.472362 4035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:11.472651 4027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:11.472898 4027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:11.474061 4027 hybrid_clock.cc:648] HybridClock initialized: now 1790479391474042 us; error 31 us; skew 500 ppm
Sep 27 03:23:11 dist-test-slave-gw2v krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:11.476755 4027 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:23:11.477173 4027 webserver.cc:460] Webserver started at http://127.0.98.1:36095/ using document root <none> and password file <none>
I20260927 03:23:11.477353 4027 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:11.477392 4027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:11.477485 4027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:11.478514 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "5e36e65ce2a440449a4e18424930f8fb"
format_stamp: "Formatted at 2026-09-27 03:23:11 on dist-test-slave-gw2v"
server_key: "9e8d5fcee451cee96989ba3428641f2b"
server_key_iv: "82d4038bd4639f7bb4b3889de6e18879"
server_key_version: "encryptionkey@0"
I20260927 03:23:11.478854 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5e36e65ce2a440449a4e18424930f8fb"
format_stamp: "Formatted at 2026-09-27 03:23:11 on dist-test-slave-gw2v"
server_key: "9e8d5fcee451cee96989ba3428641f2b"
server_key_iv: "82d4038bd4639f7bb4b3889de6e18879"
server_key_version: "encryptionkey@0"
I20260927 03:23:11.480079 4027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:11.480752 4042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:11.480906 4027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:11.480986 4027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "5e36e65ce2a440449a4e18424930f8fb"
format_stamp: "Formatted at 2026-09-27 03:23:11 on dist-test-slave-gw2v"
server_key: "9e8d5fcee451cee96989ba3428641f2b"
server_key_iv: "82d4038bd4639f7bb4b3889de6e18879"
server_key_version: "encryptionkey@0"
I20260927 03:23:11.481055 4027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:11.489668 4027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:11.491778 4027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:11.491909 4027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:11.492120 4027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:11.492398 4027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:11.492429 4027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:11.492451 4027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:11.492475 4027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:11.497992 4027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37229
I20260927 03:23:11.498031 4155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37229 every 8 connection(s)
I20260927 03:23:11.498334 4027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:23:11 dist-test-slave-gw2v krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:11.505914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 4027
I20260927 03:23:11.506007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:23:11.506170 392 external_mini_cluster.cc:1428] Setting key b4a775e4ce7be4c343a3901e024e3501
I20260927 03:23:11.507280 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:11.499074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60047 (local address 127.0.98.62:37221)
0927 03:23:11.499140 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:11.499142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:11.499353 (+ 211us) server_negotiation.cc:407] Connection header received
0927 03:23:11.500289 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:11.500290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:11.500301 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:11.500361 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:11.500957 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.501456 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.502443 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.502643 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.502695 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:11.504704 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:11.504706 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:11.504706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:11.504714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:11.505954 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:11.506323 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:11.506324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:11.506324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:11.506377 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:11.506616 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:11.506617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:11.506617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:11.506744 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:11.506808 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:11.507113 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 03:23:11.507194 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:23:11.507368 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:11.499031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37221 (local address 127.0.98.1:60047)
0927 03:23:11.499180 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:11.499221 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:11.500190 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:11.500417 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:11.500421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:11.500672 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:11.500866 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:11.500870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.501539 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.501540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:11.502369 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:11.502371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.502702 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.502703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:11.502796 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:11.503288 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:11.503296 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:11.504618 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:11.506027 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:11.506034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:11.506036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:11.506265 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:11.506429 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:11.506430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:11.506430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:11.506549 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:11.506807 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:11.506814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:11.506986 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 03:23:11.507073 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 03:23:11.507195 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260927 03:23:11.508016 4156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37221
I20260927 03:23:11.508104 4156 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:11.508279 4156 heartbeater.cc:511] Master 127.0.98.62:37221 requested a full tablet report, sending...
I20260927 03:23:11.508773 3965 ts_manager.cc:194] Registered new tserver with Master: 5e36e65ce2a440449a4e18424930f8fb (127.0.98.1:37229)
I20260927 03:23:11.509361 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60047
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:23:11.530571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37221
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:11.610970 4161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:11.611174 4161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:11.611196 4161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:11.612346 4161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:11.612388 4161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:11.612416 4161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:23:11.613876 4161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:11.614094 4161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:11.614310 4161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:23:11.619148 4161 server_base.cc:1034] running on GCE node
W20260927 03:23:11.619123 4169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:11.619094 4166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:11.619084 4167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:11.619540 4161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:11.619751 4161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:11.620886 4161 hybrid_clock.cc:648] HybridClock initialized: now 1790479391620875 us; error 28 us; skew 500 ppm
Sep 27 03:23:11 dist-test-slave-gw2v krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:11.625978 4161 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:23:11.626410 4161 webserver.cc:460] Webserver started at http://127.0.98.2:42687/ using document root <none> and password file <none>
I20260927 03:23:11.626600 4161 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:11.626638 4161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:11.626731 4161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:11.627769 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "b9d6171366504583b8ca2b4d103aff14"
format_stamp: "Formatted at 2026-09-27 03:23:11 on dist-test-slave-gw2v"
server_key: "098143991334da1350e46b97ff12bff3"
server_key_iv: "3bf1ee6f7802da1cb1850c0093ac0311"
server_key_version: "encryptionkey@0"
I20260927 03:23:11.628046 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b9d6171366504583b8ca2b4d103aff14"
format_stamp: "Formatted at 2026-09-27 03:23:11 on dist-test-slave-gw2v"
server_key: "098143991334da1350e46b97ff12bff3"
server_key_iv: "3bf1ee6f7802da1cb1850c0093ac0311"
server_key_version: "encryptionkey@0"
I20260927 03:23:11.629220 4161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:23:11.629937 4176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:11.630066 4161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:11.630137 4161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "b9d6171366504583b8ca2b4d103aff14"
format_stamp: "Formatted at 2026-09-27 03:23:11 on dist-test-slave-gw2v"
server_key: "098143991334da1350e46b97ff12bff3"
server_key_iv: "3bf1ee6f7802da1cb1850c0093ac0311"
server_key_version: "encryptionkey@0"
I20260927 03:23:11.630199 4161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:11.638808 4161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:11.640641 4161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:11.640841 4161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:11.641057 4161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:11.641366 4161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:11.641399 4161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:11.641427 4161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:11.641451 4161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:11.646919 4161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39299
I20260927 03:23:11.646986 4289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39299 every 8 connection(s)
I20260927 03:23:11.647294 4161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:23:11 dist-test-slave-gw2v krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:11.655297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 4161
I20260927 03:23:11.655409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:23:11.655582 392 external_mini_cluster.cc:1428] Setting key 23ab69b3391ef0397ace41bdd53895d9
I20260927 03:23:11.656291 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:11.647971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38471 (local address 127.0.98.62:37221)
0927 03:23:11.648052 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:11.648054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:11.648217 (+ 163us) server_negotiation.cc:407] Connection header received
0927 03:23:11.649172 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:11.649173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:11.649182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:11.649232 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:11.649839 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.650318 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.651342 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.651527 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.651578 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:11.653622 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:11.653623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:11.653624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:11.653630 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:11.654920 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:11.655291 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:11.655292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:11.655293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:11.655357 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:11.655639 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:11.655640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:11.655640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:11.655785 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:11.655851 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:11.656114 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 03:23:11.656194 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 03:23:11.656482 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:11.647936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37221 (local address 127.0.98.2:38471)
0927 03:23:11.648096 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:11.648133 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:11.649083 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:11.649292 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:11.649296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:11.649554 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:11.649774 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:11.649778 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.650395 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.650396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:11.651271 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:11.651273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.651591 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.651592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:11.651699 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:11.652172 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:11.652180 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:11.653537 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:11.654990 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:11.654995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:11.654997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:11.655214 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:11.655464 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:11.655464 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:11.655465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:11.655576 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:11.655853 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:11.655860 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:11.655995 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 03:23:11.656085 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 03:23:11.656249 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260927 03:23:11.656903 4290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37221
I20260927 03:23:11.656994 4290 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:11.657169 4290 heartbeater.cc:511] Master 127.0.98.62:37221 requested a full tablet report, sending...
I20260927 03:23:11.657531 3965 ts_manager.cc:194] Registered new tserver with Master: b9d6171366504583b8ca2b4d103aff14 (127.0.98.2:39299)
I20260927 03:23:11.657889 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38471
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:23:11.682102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37221
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:11.761893 4295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:11.762118 4295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:11.762142 4295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:11.763425 4295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:11.763489 4295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:11.763517 4295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:23:11.765303 4295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:11.765525 4295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:11.765723 4295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:11.770745 4301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:11.770792 4295 server_base.cc:1034] running on GCE node
W20260927 03:23:11.770751 4303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:11.770807 4300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:11.771212 4295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:11.771518 4295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:11.772666 4295 hybrid_clock.cc:648] HybridClock initialized: now 1790479391772650 us; error 29 us; skew 500 ppm
Sep 27 03:23:11 dist-test-slave-gw2v krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:11.775039 4295 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:23:11.775525 4295 webserver.cc:460] Webserver started at http://127.0.98.3:40025/ using document root <none> and password file <none>
I20260927 03:23:11.775743 4295 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:11.775789 4295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:11.775914 4295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:11.776999 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "cec74a99e85945f2a8a871f380705c89"
format_stamp: "Formatted at 2026-09-27 03:23:11 on dist-test-slave-gw2v"
server_key: "b169234a90aa2e121ee5caa414096c5d"
server_key_iv: "5a9d41e9141b11aaa6e4b008a5ac13cc"
server_key_version: "encryptionkey@0"
I20260927 03:23:11.777346 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cec74a99e85945f2a8a871f380705c89"
format_stamp: "Formatted at 2026-09-27 03:23:11 on dist-test-slave-gw2v"
server_key: "b169234a90aa2e121ee5caa414096c5d"
server_key_iv: "5a9d41e9141b11aaa6e4b008a5ac13cc"
server_key_version: "encryptionkey@0"
I20260927 03:23:11.778573 4295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:23:11.779451 4310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:11.779752 4295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:23:11.779855 4295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "cec74a99e85945f2a8a871f380705c89"
format_stamp: "Formatted at 2026-09-27 03:23:11 on dist-test-slave-gw2v"
server_key: "b169234a90aa2e121ee5caa414096c5d"
server_key_iv: "5a9d41e9141b11aaa6e4b008a5ac13cc"
server_key_version: "encryptionkey@0"
I20260927 03:23:11.779937 4295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:11.796860 4295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:11.798744 4295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:11.798918 4295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:11.799170 4295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:11.799450 4295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:11.799482 4295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:11.799512 4295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:11.799535 4295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:11.804881 4295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44335
I20260927 03:23:11.804921 4423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44335 every 8 connection(s)
I20260927 03:23:11.805214 4295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:23:11.807026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 4295
I20260927 03:23:11.807152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:23:11.807318 392 external_mini_cluster.cc:1428] Setting key 9b430960ba80043834cfe08e3e234677
Sep 27 03:23:11 dist-test-slave-gw2v krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:11.813912 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:11.805860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33525 (local address 127.0.98.62:37221)
0927 03:23:11.805911 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:11.805913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:11.806095 (+ 182us) server_negotiation.cc:407] Connection header received
0927 03:23:11.807074 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:11.807075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:11.807087 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:11.807144 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:11.807877 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.808476 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.809425 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.809678 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.809732 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:11.811658 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:11.811660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:11.811660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:11.811667 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:11.812899 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:11.813225 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:11.813226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:11.813227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:11.813273 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:11.813449 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:11.813450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:11.813450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:11.813543 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:11.813587 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:11.813791 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 03:23:11.813846 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:23:11.814096 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:11.805843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37221 (local address 127.0.98.3:33525)
0927 03:23:11.805982 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:11.806016 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:11.806967 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:11.807220 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:11.807225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:11.807536 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:11.807803 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:11.807808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.808553 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.808554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:11.809355 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:11.809357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.809737 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.809738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:11.809837 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:11.810324 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:11.810332 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:11.811581 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:11.812965 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:11.812970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:11.812972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:11.813167 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:11.813314 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:11.813314 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:11.813314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:11.813408 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:11.813583 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:11.813587 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:11.813696 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 03:23:11.813773 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:23:11.813883 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260927 03:23:11.814517 4424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37221
I20260927 03:23:11.814602 4424 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:11.814767 4424 heartbeater.cc:511] Master 127.0.98.62:37221 requested a full tablet report, sending...
I20260927 03:23:11.815135 3965 ts_manager.cc:194] Registered new tserver with Master: cec74a99e85945f2a8a871f380705c89 (127.0.98.3:44335)
I20260927 03:23:11.815505 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33525
I20260927 03:23:11.819162 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:23:11 dist-test-slave-gw2v krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:23:11 dist-test-slave-gw2v krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:23:11.841192 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:11.833760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.0.98.1:37229)
0927 03:23:11.833951 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:11.833956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:11.833966 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:11.833991 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:11.833993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:11.834007 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:11.834070 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:11.834673 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.835434 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.836209 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.836545 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.836595 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:11.838709 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:11.838717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:11.838717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:11.838724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:11.840184 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:11.840484 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:11.840487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:11.840488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:11.840538 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:11.840676 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:11.840677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:11.840677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:11.840804 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:11.840903 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:11.841020 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 03:23:11.841095 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":112,"threads_started":1}
Sep 27 03:23:11 dist-test-slave-gw2v krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:11.848924 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:11.842369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.0.98.62:37221)
0927 03:23:11.842412 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:11.842414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:11.842422 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:11.842465 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:11.842465 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:11.842473 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:11.842513 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:11.843039 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.843521 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.844261 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:11.844454 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:11.844506 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:11.846678 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:11.846680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:11.846680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:11.846688 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:11.847913 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:11.848208 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:11.848209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:11.848210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:11.848262 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:11.848416 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:11.848416 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:11.848416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:11.848562 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:11.848648 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:11.848758 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 03:23:11.848833 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 03:23:11.849153 3965 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40744
I20260927 03:23:11.849507 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 4027
I20260927 03:23:11.854161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 4161
I20260927 03:23:11.858031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 4295
I20260927 03:23:11.861704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 3933
2026-09-27T03:23:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1740 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:23:11 dist-test-slave-gw2v krb5kdc[4436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:23:11 dist-test-slave-gw2v krb5kdc[4436](info): set up 2 sockets
Sep 27 03:23:11 dist-test-slave-gw2v krb5kdc[4436](info): commencing operation
krb5kdc: starting...
W20260927 03:23:12.810647 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:23:12 dist-test-slave-gw2v krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:23:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:23:12.942656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35379
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:13.023973 4449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:13.024209 4449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:13.024241 4449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:13.025425 4449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:13.025467 4449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:13.025485 4449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:13.025501 4449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:13.025516 4449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:13.027113 4449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:13.027361 4449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:13.027570 4449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:13.032831 4455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:13.032861 4457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:13.032964 4449 server_base.cc:1034] running on GCE node
W20260927 03:23:13.032847 4454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:13.033315 4449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:13.033710 4449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:13.034879 4449 hybrid_clock.cc:648] HybridClock initialized: now 1790479393034863 us; error 38 us; skew 500 ppm
Sep 27 03:23:13 dist-test-slave-gw2v krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:13.037397 4449 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:23:13.037743 4449 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:13.037792 4449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:13.037890 4449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:13.039132 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "1fe4cdb7fa1f4c07894e128d5f22c85e"
format_stamp: "Formatted at 2026-09-27 03:23:13 on dist-test-slave-gw2v"
server_key: "2f4fecb04c945963a8f7e1aa26786f59"
server_key_iv: "16faefb807fe4ad75eb6486f142f7b79"
server_key_version: "encryptionkey@0"
I20260927 03:23:13.039448 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "1fe4cdb7fa1f4c07894e128d5f22c85e"
format_stamp: "Formatted at 2026-09-27 03:23:13 on dist-test-slave-gw2v"
server_key: "2f4fecb04c945963a8f7e1aa26786f59"
server_key_iv: "16faefb807fe4ad75eb6486f142f7b79"
server_key_version: "encryptionkey@0"
I20260927 03:23:13.040702 4449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:23:13.041447 4463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:13.041693 4449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:13.041778 4449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "1fe4cdb7fa1f4c07894e128d5f22c85e"
format_stamp: "Formatted at 2026-09-27 03:23:13 on dist-test-slave-gw2v"
server_key: "2f4fecb04c945963a8f7e1aa26786f59"
server_key_iv: "16faefb807fe4ad75eb6486f142f7b79"
server_key_version: "encryptionkey@0"
I20260927 03:23:13.041838 4449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:13.054597 4449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:13.056949 4449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:13.057067 4449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:13.060747 4449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35379
I20260927 03:23:13.060774 4515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35379 every 8 connection(s)
I20260927 03:23:13.061048 4449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:13.061959 4516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:13.064302 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e: Bootstrap starting.
I20260927 03:23:13.064884 4516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:13.065148 4516 log.cc:824] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:13.065783 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e: No bootstrap required, opened a new log
I20260927 03:23:13.067437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 4449
I20260927 03:23:13.067371 4516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe4cdb7fa1f4c07894e128d5f22c85e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35379 } }
I20260927 03:23:13.067517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:13.067508 4516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:13.067539 4516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe4cdb7fa1f4c07894e128d5f22c85e, State: Initialized, Role: FOLLOWER
I20260927 03:23:13.067664 392 external_mini_cluster.cc:1428] Setting key 0565c69a66be734982ddcb800c524573
I20260927 03:23:13.067683 4516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe4cdb7fa1f4c07894e128d5f22c85e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35379 } }
I20260927 03:23:13.067761 4516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:13.067798 4516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:13.067849 4516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:13.068426 4516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe4cdb7fa1f4c07894e128d5f22c85e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35379 } }
I20260927 03:23:13.068514 4516 leader_election.cc:302] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fe4cdb7fa1f4c07894e128d5f22c85e; no voters:
I20260927 03:23:13.068665 4516 leader_election.cc:288] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:13.068701 4521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:13.068847 4521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [term 1 LEADER]: Becoming Leader. State: Replica: 1fe4cdb7fa1f4c07894e128d5f22c85e, State: Running, Role: LEADER
I20260927 03:23:13.068907 4516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:13.068969 4521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe4cdb7fa1f4c07894e128d5f22c85e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35379 } }
I20260927 03:23:13.069299 4521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fe4cdb7fa1f4c07894e128d5f22c85e. Latest consensus state: current_term: 1 leader_uuid: "1fe4cdb7fa1f4c07894e128d5f22c85e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe4cdb7fa1f4c07894e128d5f22c85e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35379 } } }
I20260927 03:23:13.069504 4521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:13.069447 4523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fe4cdb7fa1f4c07894e128d5f22c85e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe4cdb7fa1f4c07894e128d5f22c85e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35379 } } }
I20260927 03:23:13.069859 4523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e [sys.catalog]: This master's current role is: LEADER
W20260927 03:23:13.070688 4537 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:23:13.070730 4537 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:23:13.070775 4530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:13.070937 4530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:13.072392 4530 catalog_manager.cc:1347] Generated new cluster ID: 3ab74f572d6044d1825d7c5363a935ea
I20260927 03:23:13.072440 4530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:23:13 dist-test-slave-gw2v krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:13.079927 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.068113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.0.98.62:35379)
0927 03:23:13.068684 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:13.068693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:13.068729 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:23:13.068853 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:13.068861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:13.068918 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:13.069883 (+ 965us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:13.069988 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.071376 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.072356 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.072952 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.073056 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.075750 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:13.075754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:13.075755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:13.075767 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:13.078267 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:13.078637 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:13.078640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:13.078641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:13.078755 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:13.078946 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:13.078947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:13.078947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:13.079331 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:13.079488 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:13.079555 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 03:23:13.079692 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":444,"threads_started":1}
I20260927 03:23:13.081053 4530 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:13.081554 4530 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:13.087038 4530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e: Generated new TSK 0
I20260927 03:23:13.087203 4530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:23:13.113924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35379
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:13.194105 4542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:13.194317 4542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:13.194340 4542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:13.195482 4542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:13.195518 4542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:13.195542 4542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:23:13.196983 4542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:13.197167 4542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:13.197353 4542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:13.201915 4550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:13.201923 4548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:13.202092 4542 server_base.cc:1034] running on GCE node
W20260927 03:23:13.201923 4547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:13.202358 4542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:13.202659 4542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:13.203815 4542 hybrid_clock.cc:648] HybridClock initialized: now 1790479393203774 us; error 52 us; skew 500 ppm
Sep 27 03:23:13 dist-test-slave-gw2v krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:13.206233 4542 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:23:13.206593 4542 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:13.206647 4542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:13.206782 4542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:13.207955 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "c5781fe7b88d46d1a6834c996a8dc91e"
format_stamp: "Formatted at 2026-09-27 03:23:13 on dist-test-slave-gw2v"
server_key: "3ef8e77e8e36c18b7c1fa35eb063523c"
server_key_iv: "6c7e1ad17354b18c36071840f9e22db9"
server_key_version: "encryptionkey@0"
I20260927 03:23:13.208328 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c5781fe7b88d46d1a6834c996a8dc91e"
format_stamp: "Formatted at 2026-09-27 03:23:13 on dist-test-slave-gw2v"
server_key: "3ef8e77e8e36c18b7c1fa35eb063523c"
server_key_iv: "6c7e1ad17354b18c36071840f9e22db9"
server_key_version: "encryptionkey@0"
I20260927 03:23:13.209806 4542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:13.210521 4556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:13.210721 4542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:13.210806 4542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "c5781fe7b88d46d1a6834c996a8dc91e"
format_stamp: "Formatted at 2026-09-27 03:23:13 on dist-test-slave-gw2v"
server_key: "3ef8e77e8e36c18b7c1fa35eb063523c"
server_key_iv: "6c7e1ad17354b18c36071840f9e22db9"
server_key_version: "encryptionkey@0"
I20260927 03:23:13.210909 4542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:13.220080 4542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:13.221809 4542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:13.221949 4542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:13.222162 4542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:13.222481 4542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:13.222514 4542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:13.222543 4542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:13.222559 4542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:13.228087 4542 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37775
I20260927 03:23:13.228148 4669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37775 every 8 connection(s)
I20260927 03:23:13.228380 4542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:23:13.228747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 4542
I20260927 03:23:13.228843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:23:13.229035 392 external_mini_cluster.cc:1428] Setting key 14d2cd54a41ceba1563589749a497816
Sep 27 03:23:13 dist-test-slave-gw2v krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 03:23:13.238142 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.229166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58805 (local address 127.0.98.62:35379)
0927 03:23:13.229228 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:13.229230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:13.229508 (+ 278us) server_negotiation.cc:407] Connection header received
0927 03:23:13.230068 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:13.230069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:13.230082 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:13.230147 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:13.231181 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.231690 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.233058 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.233255 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.233306 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.235605 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:13.235606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:13.235606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:13.235615 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:13.236981 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:13.237318 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:13.237319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:13.237319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:13.237368 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:13.237572 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:13.237572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:13.237573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:13.237677 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:13.237736 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:13.238014 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 03:23:13.238077 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:23:13.238310 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.229125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35379 (local address 127.0.98.1:58805)
0927 03:23:13.229378 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:13.229424 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:13.229955 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:13.230203 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:13.230206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:13.230459 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:13.231082 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.231089 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.231774 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.231775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.232980 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.232983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.233327 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.233328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.233494 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.233988 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:13.233998 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:13.235519 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:13.237044 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:13.237051 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:13.237053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:13.237256 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:13.237419 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:13.237420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:13.237420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:13.237520 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:13.237743 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:13.237749 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:13.237871 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 03:23:13.237993 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 03:23:13.238116 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":74,"threads_started":1}
I20260927 03:23:13.238845 4670 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35379
I20260927 03:23:13.238941 4670 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 03:23:13.239101 4670 heartbeater.cc:511] Master 127.0.98.62:35379 requested a full tablet report, sending...
I20260927 03:23:13.239591 4480 ts_manager.cc:194] Registered new tserver with Master: c5781fe7b88d46d1a6834c996a8dc91e (127.0.98.1:37775)
I20260927 03:23:13.240188 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:23:13.253273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35379
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:13.331465 4675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:13.331661 4675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:13.331683 4675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:13.332806 4675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:13.332849 4675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:13.332875 4675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:23:13.334324 4675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:13.334532 4675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:13.334739 4675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:23:13.339594 4675 server_base.cc:1034] running on GCE node
W20260927 03:23:13.339634 4683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:13.339731 4681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:13.339737 4680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:13.339985 4675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:13.340198 4675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:13.341341 4675 hybrid_clock.cc:648] HybridClock initialized: now 1790479393341328 us; error 35 us; skew 500 ppm
Sep 27 03:23:13 dist-test-slave-gw2v krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:13.343765 4675 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:23:13.344089 4675 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:13.344163 4675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:13.344257 4675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:13.345350 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "e899a6eb6ee449d68c3caf69d82483e4"
format_stamp: "Formatted at 2026-09-27 03:23:13 on dist-test-slave-gw2v"
server_key: "76a530e0525d011e1c6080a5fae118a4"
server_key_iv: "6870d5a2e5aec6ea4d0bf78d6b411983"
server_key_version: "encryptionkey@0"
I20260927 03:23:13.345680 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e899a6eb6ee449d68c3caf69d82483e4"
format_stamp: "Formatted at 2026-09-27 03:23:13 on dist-test-slave-gw2v"
server_key: "76a530e0525d011e1c6080a5fae118a4"
server_key_iv: "6870d5a2e5aec6ea4d0bf78d6b411983"
server_key_version: "encryptionkey@0"
I20260927 03:23:13.346932 4675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:23:13.347631 4689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:13.347777 4675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:13.347860 4675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "e899a6eb6ee449d68c3caf69d82483e4"
format_stamp: "Formatted at 2026-09-27 03:23:13 on dist-test-slave-gw2v"
server_key: "76a530e0525d011e1c6080a5fae118a4"
server_key_iv: "6870d5a2e5aec6ea4d0bf78d6b411983"
server_key_version: "encryptionkey@0"
I20260927 03:23:13.347913 4675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:13.358783 4675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:13.360905 4675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:13.361013 4675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:13.361188 4675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:13.361429 4675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:13.361459 4675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:13.361486 4675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:13.361503 4675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:13.366672 4675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35195
I20260927 03:23:13.366737 4802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35195 every 8 connection(s)
I20260927 03:23:13.366977 4675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:23:13.367550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 4675
I20260927 03:23:13.367645 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:23:13.367830 392 external_mini_cluster.cc:1428] Setting key 5c8f1aca78772b34364aaa8fd0cb328e
Sep 27 03:23:13 dist-test-slave-gw2v krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 03:23:13.376768 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.367624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51673 (local address 127.0.98.62:35379)
0927 03:23:13.367677 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:13.367679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:13.367896 (+ 217us) server_negotiation.cc:407] Connection header received
0927 03:23:13.368472 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:13.368473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:13.368484 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:13.368521 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:13.369721 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.370510 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.371720 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.372009 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.372085 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.374214 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:13.374216 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:13.374216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:13.374224 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:13.375663 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:13.376046 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:13.376048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:13.376048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:13.376110 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:13.376297 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:13.376297 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:13.376297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:13.376400 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:13.376443 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:13.376642 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 03:23:13.376701 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:23:13.376956 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.367562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35379 (local address 127.0.98.2:51673)
0927 03:23:13.367762 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:13.367809 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:13.368380 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:13.368590 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:13.368594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:13.368989 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:13.369648 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.369655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.370607 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.370608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.371637 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.371639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.372089 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.372090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.372190 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.372630 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:13.372638 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:13.374123 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:13.375749 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:13.375754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:13.375756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:13.375979 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:13.376170 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:13.376171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:13.376171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:13.376261 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:13.376439 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:13.376442 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:13.376560 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 03:23:13.376632 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 03:23:13.376745 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
I20260927 03:23:13.377391 4803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35379
I20260927 03:23:13.377470 4803 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:13.377614 4803 heartbeater.cc:511] Master 127.0.98.62:35379 requested a full tablet report, sending...
I20260927 03:23:13.378005 4480 ts_manager.cc:194] Registered new tserver with Master: e899a6eb6ee449d68c3caf69d82483e4 (127.0.98.2:35195)
I20260927 03:23:13.378398 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:23:13.394342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35379
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:13.473728 4808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:13.473955 4808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:13.473986 4808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:13.475211 4808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:13.475255 4808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:13.475291 4808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:23:13.476819 4808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:13.477063 4808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:13.477280 4808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:23:13.482362 4808 server_base.cc:1034] running on GCE node
W20260927 03:23:13.482345 4816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:13.482344 4814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:13.482345 4813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:13.482740 4808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:13.483017 4808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:13.484161 4808 hybrid_clock.cc:648] HybridClock initialized: now 1790479393484152 us; error 28 us; skew 500 ppm
Sep 27 03:23:13 dist-test-slave-gw2v krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:13.486629 4808 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:23:13.486992 4808 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:13.487040 4808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:13.487138 4808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:13.488160 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "48d639cc3c9149d786f4902d991dab15"
format_stamp: "Formatted at 2026-09-27 03:23:13 on dist-test-slave-gw2v"
server_key: "0c48405e8b3d4eae037e53520aeedeff"
server_key_iv: "ff59438ed0f9432a77fbda4bfc354c1a"
server_key_version: "encryptionkey@0"
I20260927 03:23:13.488444 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "48d639cc3c9149d786f4902d991dab15"
format_stamp: "Formatted at 2026-09-27 03:23:13 on dist-test-slave-gw2v"
server_key: "0c48405e8b3d4eae037e53520aeedeff"
server_key_iv: "ff59438ed0f9432a77fbda4bfc354c1a"
server_key_version: "encryptionkey@0"
I20260927 03:23:13.489673 4808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:13.490329 4822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:13.490557 4808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:13.490643 4808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "48d639cc3c9149d786f4902d991dab15"
format_stamp: "Formatted at 2026-09-27 03:23:13 on dist-test-slave-gw2v"
server_key: "0c48405e8b3d4eae037e53520aeedeff"
server_key_iv: "ff59438ed0f9432a77fbda4bfc354c1a"
server_key_version: "encryptionkey@0"
I20260927 03:23:13.490700 4808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:13.512195 4808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:13.514459 4808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:13.514573 4808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:13.514765 4808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:13.515085 4808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:13.515120 4808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:13.515141 4808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:13.515154 4808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:13.520309 4808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46307
I20260927 03:23:13.520377 4935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46307 every 8 connection(s)
I20260927 03:23:13.520596 4808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:23:13 dist-test-slave-gw2v krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:13.529210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 4808
I20260927 03:23:13.529233 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.521270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50817 (local address 127.0.98.62:35379)
0927 03:23:13.521334 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:13.521336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:13.521526 (+ 190us) server_negotiation.cc:407] Connection header received
0927 03:23:13.522025 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:13.522026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:13.522038 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:13.522088 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:13.523020 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.523514 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.524479 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.524675 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.524729 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.526715 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:13.526717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:13.526717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:13.526725 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:13.527940 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:13.528337 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:13.528339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:13.528339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:13.528396 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:13.528611 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:13.528612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:13.528612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:13.528733 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:13.528786 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:13.529055 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 03:23:13.529143 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:23:13.529301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:23:13.529398 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.521236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35379 (local address 127.0.98.3:50817)
0927 03:23:13.521408 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:13.521446 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:13.521919 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:13.522157 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:13.522162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:13.522408 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:13.522919 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.522924 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.523598 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.523599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.524391 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.524393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.524742 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.524743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.524855 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.525315 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:13.525323 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:13.526628 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:13.528033 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:13.528040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:13.528042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:13.528268 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:13.528455 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:13.528456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:13.528456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:13.528556 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:13.528784 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:13.528790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:13.528932 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 03:23:13.529026 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 03:23:13.529184 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260927 03:23:13.529485 392 external_mini_cluster.cc:1428] Setting key 26626a74a11764842954797820c4f4d5
I20260927 03:23:13.529975 4936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35379
I20260927 03:23:13.530077 4936 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:13.530234 4936 heartbeater.cc:511] Master 127.0.98.62:35379 requested a full tablet report, sending...
I20260927 03:23:13.530596 4479 ts_manager.cc:194] Registered new tserver with Master: 48d639cc3c9149d786f4902d991dab15 (127.0.98.3:46307)
I20260927 03:23:13.531011 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50817
I20260927 03:23:13.541333 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:23:13.549057 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.542682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51584 (local address 127.0.98.62:35379)
0927 03:23:13.542744 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:13.542746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:13.542866 (+ 120us) server_negotiation.cc:407] Connection header received
0927 03:23:13.542944 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:13.542944 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:13.542953 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:13.543000 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:13.543597 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.544091 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.544923 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.545115 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.545195 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.546592 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:13.546593 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:13.546594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:13.546601 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:13.547920 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:13.548256 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:13.548256 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:13.548257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:13.548306 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:13.548438 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:13.548439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:13.548439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:13.548597 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:13.548671 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:13.548911 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 03:23:13.548986 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:23:13.550959 4479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:23:13.552610 4479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:23:13.559118 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.554632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46307 (local address 127.0.0.1:45566)
0927 03:23:13.554968 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:13.555011 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:13.555088 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:13.555532 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:13.555535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:13.555539 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:13.555612 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.555614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.556895 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.556895 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.557990 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.557993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.558767 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.558768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.558876 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.558899 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 03:23:13.558971 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 03:23:13.559022 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":57,"threads_started":1}
I20260927 03:23:13.559142 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.554721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.0.98.3:46307)
0927 03:23:13.555235 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:13.555239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:13.555249 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:13.555302 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:13.555304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:13.555322 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:13.555432 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:13.555696 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.556743 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.558068 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.558602 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.558670 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.558734 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:13.559008 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 03:23:13.559055 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":130,"threads_started":1}
I20260927 03:23:13.559600 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.554634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35195 (local address 127.0.0.1:47720)
0927 03:23:13.554968 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:13.555011 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:13.555078 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:13.556118 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:13.556119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:13.556121 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:13.556176 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.556178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.557201 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.557202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.558436 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.558439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.559343 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.559345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.559419 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.559424 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:23:13.559480 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:23:13.559521 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":60,"threads_started":1}
I20260927 03:23:13.560071 4870 tablet_service.cc:1467] Processing CreateTablet for tablet 7158616c363f41889e4036f26f8e93a4 (DEFAULT_TABLE table=test-table [id=c14845e3ca8c4788ba45fa29658e2505]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:13.560298 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.554862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37775 (local address 127.0.0.1:56516)
0927 03:23:13.556032 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:13.556043 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:13.556096 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:13.556323 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:13.556324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:13.556326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:13.556364 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.556365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.557565 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.557566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.558596 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.558598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.560060 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.560060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.560105 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.560109 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:13.560153 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:23:13.560191 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1157,"thread_start_us":65,"threads_started":1}
I20260927 03:23:13.560307 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.554984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56516 (local address 127.0.98.1:37775)
0927 03:23:13.555154 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:13.555160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:13.556170 (+ 1010us) server_negotiation.cc:407] Connection header received
0927 03:23:13.556212 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:13.556215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:13.556229 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:13.556307 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:13.556506 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.557457 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.558667 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.559902 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.559973 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.560030 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:13.560179 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 03:23:13.560218 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260927 03:23:13.560421 4870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7158616c363f41889e4036f26f8e93a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:13.563068 4950 tablet_bootstrap.cc:492] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15: Bootstrap starting.
I20260927 03:23:13.563380 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.554736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47720 (local address 127.0.98.2:35195)
0927 03:23:13.555966 (+ 1230us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:13.555971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:13.555981 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:13.556014 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:13.556016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:13.556028 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:13.556118 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:13.556232 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.557103 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.558546 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.559277 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.559342 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.559399 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:13.559518 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 03:23:13.559560 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1194,"thread_start_us":1160,"threads_started":1}
I20260927 03:23:13.563750 4950 tablet_bootstrap.cc:654] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:13.563788 4604 tablet_service.cc:1467] Processing CreateTablet for tablet 7158616c363f41889e4036f26f8e93a4 (DEFAULT_TABLE table=test-table [id=c14845e3ca8c4788ba45fa29658e2505]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:13.563957 4737 tablet_service.cc:1467] Processing CreateTablet for tablet 7158616c363f41889e4036f26f8e93a4 (DEFAULT_TABLE table=test-table [id=c14845e3ca8c4788ba45fa29658e2505]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:13.564179 4604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7158616c363f41889e4036f26f8e93a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:13.564281 4737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7158616c363f41889e4036f26f8e93a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:13.564347 4950 log.cc:824] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:13.565132 4950 tablet_bootstrap.cc:492] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15: No bootstrap required, opened a new log
I20260927 03:23:13.565200 4950 ts_tablet_manager.cc:1397] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:23:13.566934 4952 tablet_bootstrap.cc:492] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4: Bootstrap starting.
I20260927 03:23:13.567147 4950 raft_consensus.cc:348] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e899a6eb6ee449d68c3caf69d82483e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "48d639cc3c9149d786f4902d991dab15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46307 } } peers { permanent_uuid: "c5781fe7b88d46d1a6834c996a8dc91e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37775 } }
I20260927 03:23:13.567322 4950 raft_consensus.cc:374] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:13.567366 4950 raft_consensus.cc:729] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d639cc3c9149d786f4902d991dab15, State: Initialized, Role: FOLLOWER
I20260927 03:23:13.567484 4950 consensus_queue.cc:260] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e899a6eb6ee449d68c3caf69d82483e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "48d639cc3c9149d786f4902d991dab15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46307 } } peers { permanent_uuid: "c5781fe7b88d46d1a6834c996a8dc91e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37775 } }
I20260927 03:23:13.567755 4950 ts_tablet_manager.cc:1428] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:23:13.567770 4952 tablet_bootstrap.cc:654] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:13.567781 4936 heartbeater.cc:503] Master 127.0.98.62:35379 was elected leader, sending a full tablet report...
I20260927 03:23:13.567946 4953 tablet_bootstrap.cc:492] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e: Bootstrap starting.
I20260927 03:23:13.568316 4952 log.cc:824] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:13.568500 4953 tablet_bootstrap.cc:654] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:13.569036 4953 log.cc:824] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:13.569232 4952 tablet_bootstrap.cc:492] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4: No bootstrap required, opened a new log
I20260927 03:23:13.569295 4952 ts_tablet_manager.cc:1397] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:23:13.569705 4953 tablet_bootstrap.cc:492] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e: No bootstrap required, opened a new log
I20260927 03:23:13.569762 4953 ts_tablet_manager.cc:1397] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:23:13.570899 4952 raft_consensus.cc:348] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e899a6eb6ee449d68c3caf69d82483e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "48d639cc3c9149d786f4902d991dab15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46307 } } peers { permanent_uuid: "c5781fe7b88d46d1a6834c996a8dc91e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37775 } }
I20260927 03:23:13.571038 4952 raft_consensus.cc:374] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:13.571069 4952 raft_consensus.cc:729] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e899a6eb6ee449d68c3caf69d82483e4, State: Initialized, Role: FOLLOWER
I20260927 03:23:13.571257 4952 consensus_queue.cc:260] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e899a6eb6ee449d68c3caf69d82483e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "48d639cc3c9149d786f4902d991dab15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46307 } } peers { permanent_uuid: "c5781fe7b88d46d1a6834c996a8dc91e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37775 } }
I20260927 03:23:13.571394 4953 raft_consensus.cc:348] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e899a6eb6ee449d68c3caf69d82483e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "48d639cc3c9149d786f4902d991dab15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46307 } } peers { permanent_uuid: "c5781fe7b88d46d1a6834c996a8dc91e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37775 } }
I20260927 03:23:13.571560 4953 raft_consensus.cc:374] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:13.571600 4953 raft_consensus.cc:729] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5781fe7b88d46d1a6834c996a8dc91e, State: Initialized, Role: FOLLOWER
I20260927 03:23:13.571627 4952 ts_tablet_manager.cc:1428] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:23:13.571714 4953 consensus_queue.cc:260] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e899a6eb6ee449d68c3caf69d82483e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "48d639cc3c9149d786f4902d991dab15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46307 } } peers { permanent_uuid: "c5781fe7b88d46d1a6834c996a8dc91e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37775 } }
I20260927 03:23:13.571772 4803 heartbeater.cc:503] Master 127.0.98.62:35379 was elected leader, sending a full tablet report...
I20260927 03:23:13.571967 4953 ts_tablet_manager.cc:1428] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:23:13.571990 4670 heartbeater.cc:503] Master 127.0.98.62:35379 was elected leader, sending a full tablet report...
W20260927 03:23:13.617479 4804 tablet.cc:2369] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:23:13.665298 4958 raft_consensus.cc:482] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:13.665453 4958 raft_consensus.cc:504] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e899a6eb6ee449d68c3caf69d82483e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "48d639cc3c9149d786f4902d991dab15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46307 } } peers { permanent_uuid: "c5781fe7b88d46d1a6834c996a8dc91e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37775 } }
I20260927 03:23:13.665928 4958 leader_election.cc:288] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e899a6eb6ee449d68c3caf69d82483e4 (127.0.98.2:35195), 48d639cc3c9149d786f4902d991dab15 (127.0.98.3:46307)
I20260927 03:23:13.670169 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.666306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46307 (local address 127.0.98.1:35739)
0927 03:23:13.666368 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:13.666381 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:13.666444 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:13.666612 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:13.666613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:13.666615 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:13.666704 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.666706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.667872 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.667874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.669110 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.669113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.669948 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.669949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.670027 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.670030 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:13.670079 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:23:13.670111 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 03:23:13.670209 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.666396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35739 (local address 127.0.98.3:46307)
0927 03:23:13.666444 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:13.666447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:13.666486 (+ 39us) server_negotiation.cc:407] Connection header received
0927 03:23:13.666516 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:13.666517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:13.666528 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:13.666589 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:13.666790 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.667760 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.669184 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.669798 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.669886 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.669899 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:13.670092 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 03:23:13.670140 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:23:13.670681 4890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7158616c363f41889e4036f26f8e93a4" candidate_uuid: "c5781fe7b88d46d1a6834c996a8dc91e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d639cc3c9149d786f4902d991dab15" is_pre_election: true
I20260927 03:23:13.670876 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.667473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38435 (local address 127.0.98.2:35195)
0927 03:23:13.667515 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:13.667517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:13.667844 (+ 327us) server_negotiation.cc:407] Connection header received
0927 03:23:13.667875 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:13.667876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:13.667889 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:13.667949 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:13.668095 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.668613 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.669898 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.670486 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.670550 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.670561 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:13.670739 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 03:23:13.670785 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:23:13.670898 4890 raft_consensus.cc:2454] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5781fe7b88d46d1a6834c996a8dc91e in term 0.
I20260927 03:23:13.671108 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.667282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35195 (local address 127.0.98.1:38435)
0927 03:23:13.667750 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:13.667760 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:13.667821 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:13.667951 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:13.667952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:13.667953 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:13.668034 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.668036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.668685 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.668686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.669828 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:13.669831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.670579 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.670580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:13.670644 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.670648 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:13.670955 (+ 307us) client_negotiation.cc:240] Negotiation successful
0927 03:23:13.670995 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":81,"threads_started":1}
I20260927 03:23:13.671123 4558 leader_election.cc:302] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48d639cc3c9149d786f4902d991dab15, c5781fe7b88d46d1a6834c996a8dc91e; no voters:
I20260927 03:23:13.671412 4958 raft_consensus.cc:2781] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:23:13.671564 4958 raft_consensus.cc:482] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:13.671599 4958 raft_consensus.cc:3037] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:13.671628 4757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7158616c363f41889e4036f26f8e93a4" candidate_uuid: "c5781fe7b88d46d1a6834c996a8dc91e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e899a6eb6ee449d68c3caf69d82483e4" is_pre_election: true
I20260927 03:23:13.671768 4757 raft_consensus.cc:2454] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5781fe7b88d46d1a6834c996a8dc91e in term 0.
I20260927 03:23:13.672516 4958 raft_consensus.cc:504] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e899a6eb6ee449d68c3caf69d82483e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "48d639cc3c9149d786f4902d991dab15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46307 } } peers { permanent_uuid: "c5781fe7b88d46d1a6834c996a8dc91e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37775 } }
I20260927 03:23:13.672657 4958 leader_election.cc:288] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [CANDIDATE]: Term 1 election: Requested vote from peers e899a6eb6ee449d68c3caf69d82483e4 (127.0.98.2:35195), 48d639cc3c9149d786f4902d991dab15 (127.0.98.3:46307)
I20260927 03:23:13.672821 4890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7158616c363f41889e4036f26f8e93a4" candidate_uuid: "c5781fe7b88d46d1a6834c996a8dc91e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d639cc3c9149d786f4902d991dab15"
I20260927 03:23:13.672904 4890 raft_consensus.cc:3037] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:13.673038 4757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7158616c363f41889e4036f26f8e93a4" candidate_uuid: "c5781fe7b88d46d1a6834c996a8dc91e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e899a6eb6ee449d68c3caf69d82483e4"
I20260927 03:23:13.673115 4757 raft_consensus.cc:3037] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:13.673919 4757 raft_consensus.cc:2454] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5781fe7b88d46d1a6834c996a8dc91e in term 1.
I20260927 03:23:13.673919 4890 raft_consensus.cc:2454] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5781fe7b88d46d1a6834c996a8dc91e in term 1.
I20260927 03:23:13.674088 4559 leader_election.cc:302] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5781fe7b88d46d1a6834c996a8dc91e, e899a6eb6ee449d68c3caf69d82483e4; no voters:
I20260927 03:23:13.674168 4958 raft_consensus.cc:2781] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:13.674293 4958 raft_consensus.cc:686] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [term 1 LEADER]: Becoming Leader. State: Replica: c5781fe7b88d46d1a6834c996a8dc91e, State: Running, Role: LEADER
I20260927 03:23:13.674391 4958 consensus_queue.cc:237] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e899a6eb6ee449d68c3caf69d82483e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "48d639cc3c9149d786f4902d991dab15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46307 } } peers { permanent_uuid: "c5781fe7b88d46d1a6834c996a8dc91e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37775 } }
I20260927 03:23:13.675330 4479 catalog_manager.cc:5495] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e reported cstate change: term changed from 0 to 1, leader changed from <none> to c5781fe7b88d46d1a6834c996a8dc91e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c5781fe7b88d46d1a6834c996a8dc91e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e899a6eb6ee449d68c3caf69d82483e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d639cc3c9149d786f4902d991dab15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5781fe7b88d46d1a6834c996a8dc91e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37775 } health_report { overall_health: HEALTHY } } }
I20260927 03:23:13.704058 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.701685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56532 (local address 127.0.98.1:37775)
0927 03:23:13.701738 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:13.701740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:13.701748 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:13.701789 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:13.701790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:13.701809 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:13.701870 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:13.702018 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.702571 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.703337 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:13.703585 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:13.703646 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:13.703762 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:13.703871 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:13.703943 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:13.703956 (+ 13us) server_negotiation.cc:299] Negotiation successful
0927 03:23:13.703975 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:23:13.706048 4890 raft_consensus.cc:1261] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15 [term 1 FOLLOWER]: Refusing update from remote peer c5781fe7b88d46d1a6834c996a8dc91e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:13.706048 4757 raft_consensus.cc:1261] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4 [term 1 FOLLOWER]: Refusing update from remote peer c5781fe7b88d46d1a6834c996a8dc91e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:13.706491 4960 consensus_queue.cc:1035] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e899a6eb6ee449d68c3caf69d82483e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:13.706619 4958 consensus_queue.cc:1035] T 7158616c363f41889e4036f26f8e93a4 P c5781fe7b88d46d1a6834c996a8dc91e [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d639cc3c9149d786f4902d991dab15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:13.709986 4964 mvcc.cc:205] Tried to move back new op lower bound from 7333803596617052160 to 7333803596491042816. Current Snapshot: MvccSnapshot[applied={T|T < 7333803596617052160}]
I20260927 03:23:13.711661 4479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:13.711766 4479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:13.712618 4479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fe4cdb7fa1f4c07894e128d5f22c85e: Sending DeleteTablet for 3 replicas of tablet 7158616c363f41889e4036f26f8e93a4
I20260927 03:23:13.712988 4870 tablet_service.cc:1514] Processing DeleteTablet for tablet 7158616c363f41889e4036f26f8e93a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45566
I20260927 03:23:13.712991 4737 tablet_service.cc:1514] Processing DeleteTablet for tablet 7158616c363f41889e4036f26f8e93a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47720
I20260927 03:23:13.713016 4604 tablet_service.cc:1514] Processing DeleteTablet for tablet 7158616c363f41889e4036f26f8e93a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56516
I20260927 03:23:13.713193 4973 tablet_replica.cc:330] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4: stopping tablet replica
I20260927 03:23:13.713294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 4542
I20260927 03:23:13.713331 4973 raft_consensus.cc:2229] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:13.713451 4973 raft_consensus.cc:2258] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:13.713814 4972 tablet_replica.cc:330] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15: stopping tablet replica
I20260927 03:23:13.713907 4972 raft_consensus.cc:2229] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:13.713999 4972 raft_consensus.cc:2258] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:13.714417 4972 ts_tablet_manager.cc:1905] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:13.714843 4973 ts_tablet_manager.cc:1905] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:13.716194 4973 ts_tablet_manager.cc:1918] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:13.716218 4972 ts_tablet_manager.cc:1918] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:13.716274 4973 log.cc:1196] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/7158616c363f41889e4036f26f8e93a4
I20260927 03:23:13.716307 4972 log.cc:1196] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/7158616c363f41889e4036f26f8e93a4
I20260927 03:23:13.716552 4973 ts_tablet_manager.cc:1939] T 7158616c363f41889e4036f26f8e93a4 P e899a6eb6ee449d68c3caf69d82483e4: Deleting consensus metadata
I20260927 03:23:13.716609 4972 ts_tablet_manager.cc:1939] T 7158616c363f41889e4036f26f8e93a4 P 48d639cc3c9149d786f4902d991dab15: Deleting consensus metadata
I20260927 03:23:13.716961 4465 catalog_manager.cc:4841] TS 48d639cc3c9149d786f4902d991dab15 (127.0.98.3:46307): tablet 7158616c363f41889e4036f26f8e93a4 (table test-table [id=c14845e3ca8c4788ba45fa29658e2505]) successfully deleted
I20260927 03:23:13.716974 4466 catalog_manager.cc:4841] TS e899a6eb6ee449d68c3caf69d82483e4 (127.0.98.2:35195): tablet 7158616c363f41889e4036f26f8e93a4 (table test-table [id=c14845e3ca8c4788ba45fa29658e2505]) successfully deleted
W20260927 03:23:13.719409 4466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37775 (error 108)
I20260927 03:23:13.719587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 4675
I20260927 03:23:13.720141 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.719713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37775 (local address 127.0.0.1:56536)
0927 03:23:13.720073 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:13.720110 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340}
W20260927 03:23:13.720309 4466 catalog_manager.cc:4584] TS c5781fe7b88d46d1a6834c996a8dc91e (127.0.98.1:37775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7158616c363f41889e4036f26f8e93a4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37775: connect: Connection refused (error 111)
I20260927 03:23:13.724015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 4808
I20260927 03:23:13.727725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 4449
2026-09-27T03:23:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1866 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T03:23:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:13Z Disabled control of system clock
I20260927 03:23:13.753123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43171
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:23:13.832073 4979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:13.832304 4979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:13.832325 4979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:13.833462 4979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:13.833499 4979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:13.833510 4979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:13.833521 4979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:13.833532 4979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:13.834882 4979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:13.835098 4979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:13.835350 4979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:13.840407 4987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:13.840338 4985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:13.840528 4979 server_base.cc:1034] running on GCE node
W20260927 03:23:13.840338 4984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:13.840970 4979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:13.841279 4979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:13.842456 4979 hybrid_clock.cc:648] HybridClock initialized: now 1790479393842436 us; error 34 us; skew 500 ppm
I20260927 03:23:13.843832 4979 webserver.cc:460] Webserver started at http://127.0.98.62:46309/ using document root <none> and password file <none>
I20260927 03:23:13.844090 4979 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:13.844138 4979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:13.844256 4979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:13.845324 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "52e815688f2d46059f4afa4a32cea25c"
format_stamp: "Formatted at 2026-09-27 03:23:13 on dist-test-slave-gw2v"
server_key: "eb5f38b81fc297ecc7fcebaeb567302b"
server_key_iv: "a0bf5ef9acef84c874b2565dc6d744a6"
server_key_version: "encryptionkey@0"
I20260927 03:23:13.845695 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "52e815688f2d46059f4afa4a32cea25c"
format_stamp: "Formatted at 2026-09-27 03:23:13 on dist-test-slave-gw2v"
server_key: "eb5f38b81fc297ecc7fcebaeb567302b"
server_key_iv: "a0bf5ef9acef84c874b2565dc6d744a6"
server_key_version: "encryptionkey@0"
I20260927 03:23:13.847088 4979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:23:13.847838 4993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:13.847986 4979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:13.848066 4979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "52e815688f2d46059f4afa4a32cea25c"
format_stamp: "Formatted at 2026-09-27 03:23:13 on dist-test-slave-gw2v"
server_key: "eb5f38b81fc297ecc7fcebaeb567302b"
server_key_iv: "a0bf5ef9acef84c874b2565dc6d744a6"
server_key_version: "encryptionkey@0"
I20260927 03:23:13.848137 4979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:13.851648 4979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:13.851926 4979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:13.852043 4979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:13.855734 4979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43171
I20260927 03:23:13.855764 5045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43171 every 8 connection(s)
I20260927 03:23:13.856128 4979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:13.856756 5046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:13.856953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 4979
I20260927 03:23:13.857028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:13.857160 392 external_mini_cluster.cc:1428] Setting key c175129235e8bdc6edd6c1849f4d1a01
I20260927 03:23:13.859891 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:13.857731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.0.98.62:43171)
0927 03:23:13.858063 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:13.858071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:13.858096 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:23:13.858541 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:13.858549 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:13.858597 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:13.858729 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:13.859478 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:13.859481 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:13.859481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:13.859485 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:13.859527 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:13.859538 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:13.859597 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:13.859658 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 03:23:13.859726 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":215,"threads_started":1}
I20260927 03:23:13.860723 5046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c: Bootstrap starting.
I20260927 03:23:13.861318 5046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:13.861598 5046 log.cc:824] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:13.862277 5046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c: No bootstrap required, opened a new log
I20260927 03:23:13.863631 5046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e815688f2d46059f4afa4a32cea25c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43171 } }
I20260927 03:23:13.863787 5046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:13.863832 5046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e815688f2d46059f4afa4a32cea25c, State: Initialized, Role: FOLLOWER
I20260927 03:23:13.863955 5046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e815688f2d46059f4afa4a32cea25c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43171 } }
I20260927 03:23:13.864027 5046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:13.864065 5046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:13.864113 5046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:13.864635 5046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e815688f2d46059f4afa4a32cea25c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43171 } }
I20260927 03:23:13.864733 5046 leader_election.cc:302] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52e815688f2d46059f4afa4a32cea25c; no voters:
I20260927 03:23:13.864907 5046 leader_election.cc:288] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:13.865195 5046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:13.865226 5051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:13.865342 5051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [term 1 LEADER]: Becoming Leader. State: Replica: 52e815688f2d46059f4afa4a32cea25c, State: Running, Role: LEADER
I20260927 03:23:13.865425 5051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e815688f2d46059f4afa4a32cea25c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43171 } }
I20260927 03:23:13.865810 5053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52e815688f2d46059f4afa4a32cea25c. Latest consensus state: current_term: 1 leader_uuid: "52e815688f2d46059f4afa4a32cea25c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e815688f2d46059f4afa4a32cea25c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43171 } } }
I20260927 03:23:13.865916 5053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:13.866094 5052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52e815688f2d46059f4afa4a32cea25c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e815688f2d46059f4afa4a32cea25c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43171 } } }
I20260927 03:23:13.866163 5052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:13.866139 5060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:13.866709 5060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:13.868204 5060 catalog_manager.cc:1347] Generated new cluster ID: 393f5f8f604d478b8113a955096a60e5
I20260927 03:23:13.868255 5060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:23:13.877662 5060 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:13.877846 5060 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:13.881968 5060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c: Generated new TSK 0
I20260927 03:23:13.882153 5060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:23:13.889737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43171
--builtin_ntp_servers=127.0.98.20:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:23:13.995857 5070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:13.996071 5070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:13.996109 5070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:13.997299 5070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:13.997344 5070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:13.997375 5070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:23:13.998744 5070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:13.999009 5070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:13.999290 5070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:14.005060 5078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:14.005141 5075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:14.005064 5076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:14.005851 5070 server_base.cc:1034] running on GCE node
I20260927 03:23:14.006179 5070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:14.006488 5070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:14.007637 5070 hybrid_clock.cc:648] HybridClock initialized: now 1790479394007610 us; error 41 us; skew 500 ppm
I20260927 03:23:14.009161 5070 webserver.cc:460] Webserver started at http://127.0.98.1:33619/ using document root <none> and password file <none>
I20260927 03:23:14.009445 5070 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:14.009510 5070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:14.009665 5070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:14.011143 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "f07637a65b53419db4d1b3b39b30e17f"
format_stamp: "Formatted at 2026-09-27 03:23:14 on dist-test-slave-gw2v"
server_key: "ed9f0283db4d86a4dbbb92ceb0c20523"
server_key_iv: "80a2a24e9215b338dea3f08f24b20d4c"
server_key_version: "encryptionkey@0"
I20260927 03:23:14.011554 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f07637a65b53419db4d1b3b39b30e17f"
format_stamp: "Formatted at 2026-09-27 03:23:14 on dist-test-slave-gw2v"
server_key: "ed9f0283db4d86a4dbbb92ceb0c20523"
server_key_iv: "80a2a24e9215b338dea3f08f24b20d4c"
server_key_version: "encryptionkey@0"
I20260927 03:23:14.013082 5070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:23:14.013942 5084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:14.014123 5070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:14.014209 5070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "f07637a65b53419db4d1b3b39b30e17f"
format_stamp: "Formatted at 2026-09-27 03:23:14 on dist-test-slave-gw2v"
server_key: "ed9f0283db4d86a4dbbb92ceb0c20523"
server_key_iv: "80a2a24e9215b338dea3f08f24b20d4c"
server_key_version: "encryptionkey@0"
I20260927 03:23:14.014276 5070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:14.019059 5070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:14.019348 5070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:14.019470 5070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:14.019659 5070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:14.020004 5070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:14.020048 5070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:14.020073 5070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:14.020095 5070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:14.025780 5070 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38543
I20260927 03:23:14.025820 5197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38543 every 8 connection(s)
I20260927 03:23:14.026139 5070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:23:14.028177 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 5070
I20260927 03:23:14.028280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:23:14.028451 392 external_mini_cluster.cc:1428] Setting key c7b528a9f167ac8ef191b8e49ae82f09
I20260927 03:23:14.029317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43171
--builtin_ntp_servers=127.0.98.20:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 03:23:14.029323 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.026887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48985 (local address 127.0.98.62:43171)
0927 03:23:14.026949 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:14.026951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:14.027186 (+ 235us) server_negotiation.cc:407] Connection header received
0927 03:23:14.027754 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:14.027755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:14.027761 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:14.027814 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:14.029106 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:14.029107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:14.029107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:14.029112 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:14.029126 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:14.029131 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:14.029172 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:14.029242 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 03:23:14.029255 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:23:14.029527 5200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.026856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43171 (local address 127.0.98.1:48985)
0927 03:23:14.027046 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:14.027094 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:14.027633 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:14.027898 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:14.027902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:14.027911 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:14.028954 (+ 1043us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:23:14.028954 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:14.028996 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:23:14.029035 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:14.029177 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:14.029182 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:14.029185 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:14.029254 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:23:14.029316 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260927 03:23:14.030798 5198 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43171
I20260927 03:23:14.030925 5198 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:14.031085 5198 heartbeater.cc:511] Master 127.0.98.62:43171 requested a full tablet report, sending...
I20260927 03:23:14.031596 5010 ts_manager.cc:194] Registered new tserver with Master: f07637a65b53419db4d1b3b39b30e17f (127.0.98.1:38543)
W20260927 03:23:14.109951 5201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:14.110188 5201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:14.110219 5201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:14.111431 5201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:14.111474 5201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:14.111508 5201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:23:14.112840 5201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:14.113080 5201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:14.113350 5201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:14.118230 5209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:14.118242 5206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:14.118393 5201 server_base.cc:1034] running on GCE node
W20260927 03:23:14.118243 5207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:14.118737 5201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:14.118993 5201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:14.120172 5201 hybrid_clock.cc:648] HybridClock initialized: now 1790479394120151 us; error 35 us; skew 500 ppm
I20260927 03:23:14.121413 5201 webserver.cc:460] Webserver started at http://127.0.98.2:45417/ using document root <none> and password file <none>
I20260927 03:23:14.121641 5201 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:14.121690 5201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:14.121817 5201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:14.122910 5201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "39b222b883cc4d7e9ae8a0d1a5644706"
format_stamp: "Formatted at 2026-09-27 03:23:14 on dist-test-slave-gw2v"
server_key: "279e562e4897492b3697a5eed4be3298"
server_key_iv: "e5ddacfa53ed061738e42135ddccdfb1"
server_key_version: "encryptionkey@0"
I20260927 03:23:14.123256 5201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "39b222b883cc4d7e9ae8a0d1a5644706"
format_stamp: "Formatted at 2026-09-27 03:23:14 on dist-test-slave-gw2v"
server_key: "279e562e4897492b3697a5eed4be3298"
server_key_iv: "e5ddacfa53ed061738e42135ddccdfb1"
server_key_version: "encryptionkey@0"
I20260927 03:23:14.124584 5201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:14.125329 5215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:14.125586 5201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:14.125675 5201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "39b222b883cc4d7e9ae8a0d1a5644706"
format_stamp: "Formatted at 2026-09-27 03:23:14 on dist-test-slave-gw2v"
server_key: "279e562e4897492b3697a5eed4be3298"
server_key_iv: "e5ddacfa53ed061738e42135ddccdfb1"
server_key_version: "encryptionkey@0"
I20260927 03:23:14.125736 5201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:14.129117 5201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:14.129382 5201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:14.129508 5201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:14.129688 5201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:14.129987 5201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:14.130024 5201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:14.130051 5201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:14.130074 5201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:14.135596 5201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42069
I20260927 03:23:14.135658 5328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42069 every 8 connection(s)
I20260927 03:23:14.135965 5201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:23:14.138944 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.136739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44355 (local address 127.0.98.62:43171)
0927 03:23:14.136802 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:14.136804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:14.137004 (+ 200us) server_negotiation.cc:407] Connection header received
0927 03:23:14.137461 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:14.137462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:14.137468 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:14.137534 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:14.138667 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:14.138669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:14.138669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:14.138674 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:14.138693 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:14.138698 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:14.138757 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:14.138810 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 03:23:14.138855 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:23:14.139133 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.136703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43171 (local address 127.0.98.2:44355)
0927 03:23:14.136885 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:14.136920 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:14.137375 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:14.137599 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:14.137603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:14.137612 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:14.138485 (+ 873us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:23:14.138486 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:14.138526 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:23:14.138562 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:14.138760 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:14.138763 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:14.138765 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:23:14.138876 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 03:23:14.138928 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260927 03:23:14.139647 5329 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43171
I20260927 03:23:14.139736 5329 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:14.139883 5329 heartbeater.cc:511] Master 127.0.98.62:43171 requested a full tablet report, sending...
I20260927 03:23:14.140215 5010 ts_manager.cc:194] Registered new tserver with Master: 39b222b883cc4d7e9ae8a0d1a5644706 (127.0.98.2:42069)
I20260927 03:23:14.144291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 5201
I20260927 03:23:14.144363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:23:14.144501 392 external_mini_cluster.cc:1428] Setting key 0db47c0462bd63011cbd8fc4fe9418b2
I20260927 03:23:14.145345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43171
--builtin_ntp_servers=127.0.98.20:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:23:14.223932 5332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:14.224138 5332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:14.224166 5332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:14.225313 5332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:14.225353 5332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:14.225385 5332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:23:14.226698 5332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:14.226948 5332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:14.227216 5332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:14.231948 5337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:14.231954 5338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:14.232019 5332 server_base.cc:1034] running on GCE node
W20260927 03:23:14.231943 5340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:14.232350 5332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:14.232602 5332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:14.233757 5332 hybrid_clock.cc:648] HybridClock initialized: now 1790479394233745 us; error 30 us; skew 500 ppm
I20260927 03:23:14.235066 5332 webserver.cc:460] Webserver started at http://127.0.98.3:36409/ using document root <none> and password file <none>
I20260927 03:23:14.235297 5332 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:14.235344 5332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:14.235540 5332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:14.236620 5332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0"
format_stamp: "Formatted at 2026-09-27 03:23:14 on dist-test-slave-gw2v"
server_key: "23f6e84b1decda99da1fccf5b64a2465"
server_key_iv: "7bf2fe385bb35b53fb4321b2f4dc856e"
server_key_version: "encryptionkey@0"
I20260927 03:23:14.236984 5332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0"
format_stamp: "Formatted at 2026-09-27 03:23:14 on dist-test-slave-gw2v"
server_key: "23f6e84b1decda99da1fccf5b64a2465"
server_key_iv: "7bf2fe385bb35b53fb4321b2f4dc856e"
server_key_version: "encryptionkey@0"
I20260927 03:23:14.238164 5332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:14.238933 5346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:14.239070 5332 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:14.239149 5332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0"
format_stamp: "Formatted at 2026-09-27 03:23:14 on dist-test-slave-gw2v"
server_key: "23f6e84b1decda99da1fccf5b64a2465"
server_key_iv: "7bf2fe385bb35b53fb4321b2f4dc856e"
server_key_version: "encryptionkey@0"
I20260927 03:23:14.239216 5332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:14.242523 5332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:14.242765 5332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:14.242906 5332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:14.243094 5332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:14.243372 5332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:14.243412 5332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:14.243432 5332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:14.243455 5332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:14.248844 5332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42847
I20260927 03:23:14.248880 5459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42847 every 8 connection(s)
I20260927 03:23:14.249189 5332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:23:14.249706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 5332
I20260927 03:23:14.249790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:23:14.249985 392 external_mini_cluster.cc:1428] Setting key 09dcc26137c6f0b3f035e6df9c600e4f
I20260927 03:23:14.252285 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.249866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55051 (local address 127.0.98.62:43171)
0927 03:23:14.249920 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:14.249922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:14.250207 (+ 285us) server_negotiation.cc:407] Connection header received
0927 03:23:14.250922 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:14.250924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:14.250930 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:14.250968 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:14.252041 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:14.252042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:14.252043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:14.252047 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:14.252063 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:14.252068 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:14.252115 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:14.252200 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:23:14.252214 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:23:14.252488 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.249847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43171 (local address 127.0.98.3:55051)
0927 03:23:14.250037 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:14.250110 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:14.250760 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:14.251055 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:14.251059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:14.251069 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:14.251902 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:23:14.251902 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:14.251933 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:23:14.251962 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:14.252128 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:14.252133 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:14.252136 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:14.252204 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 03:23:14.252269 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260927 03:23:14.252923 5460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43171
I20260927 03:23:14.253008 5460 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:14.253123 5460 heartbeater.cc:511] Master 127.0.98.62:43171 requested a full tablet report, sending...
I20260927 03:23:14.253469 5010 ts_manager.cc:194] Registered new tserver with Master: 0d3246efd3cb48d5b0fcadf5ad1b6db0 (127.0.98.3:42847)
I20260927 03:23:14.261579 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:23:14.264019 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.262933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.0.98.62:43171)
0927 03:23:14.262982 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:14.262984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:14.263080 (+ 96us) server_negotiation.cc:407] Connection header received
0927 03:23:14.263149 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:14.263150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:14.263156 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:14.263200 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:14.263773 (+ 573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:14.263774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:14.263774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:14.263779 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:14.263794 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:14.263798 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:14.263852 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:14.263927 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 03:23:14.263940 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:23:14.265429 5010 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:23:14.267074 5010 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:23:14.270694 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.269083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42847 (local address 127.0.0.1:54466)
0927 03:23:14.269374 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:14.269384 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:14.269427 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:14.269681 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:14.269687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:14.269694 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:14.270154 (+ 460us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:23:14.270155 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:14.270172 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:23:14.270196 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:14.270550 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:14.270551 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:14.270551 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 03:23:14.270603 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:23:14.270616 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260927 03:23:14.270704 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.269243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54466 (local address 127.0.98.3:42847)
0927 03:23:14.269555 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:14.269559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:14.269567 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:14.269589 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:14.269591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:14.269600 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:14.269674 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:14.270452 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:14.270455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:14.270455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:14.270459 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:14.270483 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:14.270495 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:14.270552 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:14.270611 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 03:23:14.270624 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":80,"threads_started":1}
I20260927 03:23:14.270763 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.269167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.0.98.1:38543)
0927 03:23:14.269363 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:14.269368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:14.269378 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:14.269403 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:14.269405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:14.269415 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:14.269479 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:14.270349 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:14.270360 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:14.270360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:14.270365 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:14.270401 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:14.270430 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:14.270487 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:14.270664 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 03:23:14.270680 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20260927 03:23:14.270785 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.269083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38543 (local address 127.0.0.1:47030)
0927 03:23:14.269221 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:14.269264 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:14.269321 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:14.269815 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:14.269816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:14.269818 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:14.270263 (+ 445us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:23:14.270264 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:14.270269 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:23:14.270279 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:14.270489 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:14.270496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:14.270498 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:23:14.270558 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:23:14.270571 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":51,"threads_started":1}
I20260927 03:23:14.271466 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.269258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42069 (local address 127.0.0.1:39300)
0927 03:23:14.270294 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:14.270303 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:14.270348 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:14.270738 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:14.270739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:14.270741 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:14.271108 (+ 367us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:23:14.271108 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:14.271115 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:23:14.271126 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:14.271344 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:14.271345 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:14.271345 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 03:23:14.271391 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:23:14.271403 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"thread_start_us":67,"threads_started":1}
I20260927 03:23:14.271502 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.269485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39300 (local address 127.0.98.2:42069)
0927 03:23:14.269667 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:14.269681 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:14.270389 (+ 708us) server_negotiation.cc:407] Connection header received
0927 03:23:14.270417 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:14.270419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:14.270426 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:14.270485 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:14.271207 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:14.271214 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:14.271214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:14.271218 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:14.271256 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:14.271285 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:14.271342 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:14.271395 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 03:23:14.271409 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260927 03:23:14.271471 5132 tablet_service.cc:1467] Processing CreateTablet for tablet 2c63fedc66ac404992d465cd6b7cd8e1 (DEFAULT_TABLE table=test-table [id=a21a99617470435a83004fb65ed3272d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:14.271874 5132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c63fedc66ac404992d465cd6b7cd8e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:14.272136 5263 tablet_service.cc:1467] Processing CreateTablet for tablet 2c63fedc66ac404992d465cd6b7cd8e1 (DEFAULT_TABLE table=test-table [id=a21a99617470435a83004fb65ed3272d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:14.272234 5394 tablet_service.cc:1467] Processing CreateTablet for tablet 2c63fedc66ac404992d465cd6b7cd8e1 (DEFAULT_TABLE table=test-table [id=a21a99617470435a83004fb65ed3272d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:14.272472 5394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c63fedc66ac404992d465cd6b7cd8e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:14.272503 5263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c63fedc66ac404992d465cd6b7cd8e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:14.274899 5474 tablet_bootstrap.cc:492] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0: Bootstrap starting.
I20260927 03:23:14.275318 5475 tablet_bootstrap.cc:492] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f: Bootstrap starting.
I20260927 03:23:14.275513 5474 tablet_bootstrap.cc:654] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:14.275538 5476 tablet_bootstrap.cc:492] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706: Bootstrap starting.
I20260927 03:23:14.275813 5474 log.cc:824] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:14.275964 5475 tablet_bootstrap.cc:654] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:14.276189 5476 tablet_bootstrap.cc:654] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:14.276258 5475 log.cc:824] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:14.276427 5476 log.cc:824] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:14.276917 5475 tablet_bootstrap.cc:492] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f: No bootstrap required, opened a new log
I20260927 03:23:14.276981 5475 ts_tablet_manager.cc:1397] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:23:14.277129 5474 tablet_bootstrap.cc:492] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0: No bootstrap required, opened a new log
I20260927 03:23:14.277148 5476 tablet_bootstrap.cc:492] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706: No bootstrap required, opened a new log
I20260927 03:23:14.277175 5474 ts_tablet_manager.cc:1397] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:23:14.277191 5476 ts_tablet_manager.cc:1397] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:23:14.278942 5476 raft_consensus.cc:348] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42847 } } peers { permanent_uuid: "f07637a65b53419db4d1b3b39b30e17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38543 } } peers { permanent_uuid: "39b222b883cc4d7e9ae8a0d1a5644706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42069 } }
I20260927 03:23:14.279073 5476 raft_consensus.cc:374] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:14.279094 5476 raft_consensus.cc:729] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b222b883cc4d7e9ae8a0d1a5644706, State: Initialized, Role: FOLLOWER
I20260927 03:23:14.279183 5476 consensus_queue.cc:260] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42847 } } peers { permanent_uuid: "f07637a65b53419db4d1b3b39b30e17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38543 } } peers { permanent_uuid: "39b222b883cc4d7e9ae8a0d1a5644706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42069 } }
I20260927 03:23:14.279399 5476 ts_tablet_manager.cc:1428] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:23:14.279439 5329 heartbeater.cc:503] Master 127.0.98.62:43171 was elected leader, sending a full tablet report...
I20260927 03:23:14.279453 5475 raft_consensus.cc:348] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42847 } } peers { permanent_uuid: "f07637a65b53419db4d1b3b39b30e17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38543 } } peers { permanent_uuid: "39b222b883cc4d7e9ae8a0d1a5644706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42069 } }
I20260927 03:23:14.279603 5475 raft_consensus.cc:374] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:14.279639 5475 raft_consensus.cc:729] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07637a65b53419db4d1b3b39b30e17f, State: Initialized, Role: FOLLOWER
I20260927 03:23:14.279749 5475 consensus_queue.cc:260] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42847 } } peers { permanent_uuid: "f07637a65b53419db4d1b3b39b30e17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38543 } } peers { permanent_uuid: "39b222b883cc4d7e9ae8a0d1a5644706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42069 } }
I20260927 03:23:14.279992 5475 ts_tablet_manager.cc:1428] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:23:14.279928 5474 raft_consensus.cc:348] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42847 } } peers { permanent_uuid: "f07637a65b53419db4d1b3b39b30e17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38543 } } peers { permanent_uuid: "39b222b883cc4d7e9ae8a0d1a5644706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42069 } }
I20260927 03:23:14.280026 5198 heartbeater.cc:503] Master 127.0.98.62:43171 was elected leader, sending a full tablet report...
I20260927 03:23:14.280061 5474 raft_consensus.cc:374] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:14.280092 5474 raft_consensus.cc:729] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3246efd3cb48d5b0fcadf5ad1b6db0, State: Initialized, Role: FOLLOWER
I20260927 03:23:14.280208 5474 consensus_queue.cc:260] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42847 } } peers { permanent_uuid: "f07637a65b53419db4d1b3b39b30e17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38543 } } peers { permanent_uuid: "39b222b883cc4d7e9ae8a0d1a5644706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42069 } }
I20260927 03:23:14.280447 5474 ts_tablet_manager.cc:1428] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260927 03:23:14.280478 5460 heartbeater.cc:503] Master 127.0.98.62:43171 was elected leader, sending a full tablet report...
W20260927 03:23:14.386588 5330 tablet.cc:2369] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:23:14.472558 5481 raft_consensus.cc:482] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:14.472702 5481 raft_consensus.cc:504] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42847 } } peers { permanent_uuid: "f07637a65b53419db4d1b3b39b30e17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38543 } } peers { permanent_uuid: "39b222b883cc4d7e9ae8a0d1a5644706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42069 } }
I20260927 03:23:14.473160 5481 leader_election.cc:288] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d3246efd3cb48d5b0fcadf5ad1b6db0 (127.0.98.3:42847), 39b222b883cc4d7e9ae8a0d1a5644706 (127.0.98.2:42069)
I20260927 03:23:14.474382 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.473443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51527 (local address 127.0.98.2:42069)
0927 03:23:14.473486 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:14.473488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:14.473507 (+ 19us) server_negotiation.cc:407] Connection header received
0927 03:23:14.473535 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:14.473535 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:14.473540 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:14.473595 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:14.474154 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:14.474156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:14.474156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:14.474161 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:14.474178 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:14.474182 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:14.474237 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:14.474298 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 03:23:14.474311 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:23:14.474361 5200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.473354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42069 (local address 127.0.98.1:51527)
0927 03:23:14.473409 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:14.473420 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:14.473462 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:14.473617 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:14.473619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:14.473622 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:14.474053 (+ 431us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:23:14.474053 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:14.474060 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:23:14.474070 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:14.474239 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:14.474240 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:14.474241 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:23:14.474284 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:23:14.474296 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 03:23:14.474448 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.473423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38191 (local address 127.0.98.3:42847)
0927 03:23:14.473519 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:14.473521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:14.473750 (+ 229us) server_negotiation.cc:407] Connection header received
0927 03:23:14.473783 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:14.473784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:14.473790 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:14.473845 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:14.474250 (+ 405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:14.474251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:14.474251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:14.474255 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:14.474269 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:14.474273 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:14.474324 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:14.474369 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 03:23:14.474381 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 03:23:14.474524 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.473356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42847 (local address 127.0.98.1:38191)
0927 03:23:14.473671 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:14.473680 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:14.473738 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:14.473853 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:14.473853 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:14.473855 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:14.474166 (+ 311us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:23:14.474167 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:14.474171 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:23:14.474179 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:14.474325 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:14.474326 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:14.474326 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 03:23:14.474368 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 03:23:14.474378 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":154,"thread_start_us":95,"threads_started":1}
I20260927 03:23:14.474745 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c63fedc66ac404992d465cd6b7cd8e1" candidate_uuid: "f07637a65b53419db4d1b3b39b30e17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b222b883cc4d7e9ae8a0d1a5644706" is_pre_election: true
I20260927 03:23:14.474865 5414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c63fedc66ac404992d465cd6b7cd8e1" candidate_uuid: "f07637a65b53419db4d1b3b39b30e17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0" is_pre_election: true
I20260927 03:23:14.474931 5283 raft_consensus.cc:2454] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f07637a65b53419db4d1b3b39b30e17f in term 0.
I20260927 03:23:14.474998 5414 raft_consensus.cc:2454] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f07637a65b53419db4d1b3b39b30e17f in term 0.
I20260927 03:23:14.475184 5085 leader_election.cc:302] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b222b883cc4d7e9ae8a0d1a5644706, f07637a65b53419db4d1b3b39b30e17f; no voters:
I20260927 03:23:14.475302 5481 raft_consensus.cc:2781] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:23:14.475361 5481 raft_consensus.cc:482] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:14.475380 5481 raft_consensus.cc:3037] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:14.476061 5481 raft_consensus.cc:504] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42847 } } peers { permanent_uuid: "f07637a65b53419db4d1b3b39b30e17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38543 } } peers { permanent_uuid: "39b222b883cc4d7e9ae8a0d1a5644706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42069 } }
I20260927 03:23:14.476171 5481 leader_election.cc:288] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [CANDIDATE]: Term 1 election: Requested vote from peers 0d3246efd3cb48d5b0fcadf5ad1b6db0 (127.0.98.3:42847), 39b222b883cc4d7e9ae8a0d1a5644706 (127.0.98.2:42069)
I20260927 03:23:14.476313 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c63fedc66ac404992d465cd6b7cd8e1" candidate_uuid: "f07637a65b53419db4d1b3b39b30e17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b222b883cc4d7e9ae8a0d1a5644706"
I20260927 03:23:14.476363 5414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c63fedc66ac404992d465cd6b7cd8e1" candidate_uuid: "f07637a65b53419db4d1b3b39b30e17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0"
I20260927 03:23:14.476390 5283 raft_consensus.cc:3037] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:14.476457 5414 raft_consensus.cc:3037] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:14.477082 5283 raft_consensus.cc:2454] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f07637a65b53419db4d1b3b39b30e17f in term 1.
I20260927 03:23:14.477210 5414 raft_consensus.cc:2454] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f07637a65b53419db4d1b3b39b30e17f in term 1.
I20260927 03:23:14.477242 5085 leader_election.cc:302] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b222b883cc4d7e9ae8a0d1a5644706, f07637a65b53419db4d1b3b39b30e17f; no voters:
I20260927 03:23:14.477414 5481 raft_consensus.cc:2781] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:14.477533 5481 raft_consensus.cc:686] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [term 1 LEADER]: Becoming Leader. State: Replica: f07637a65b53419db4d1b3b39b30e17f, State: Running, Role: LEADER
I20260927 03:23:14.477640 5481 consensus_queue.cc:237] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42847 } } peers { permanent_uuid: "f07637a65b53419db4d1b3b39b30e17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38543 } } peers { permanent_uuid: "39b222b883cc4d7e9ae8a0d1a5644706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42069 } }
I20260927 03:23:14.478542 5009 catalog_manager.cc:5495] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f reported cstate change: term changed from 0 to 1, leader changed from <none> to f07637a65b53419db4d1b3b39b30e17f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f07637a65b53419db4d1b3b39b30e17f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f07637a65b53419db4d1b3b39b30e17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39b222b883cc4d7e9ae8a0d1a5644706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42069 } health_report { overall_health: UNKNOWN } } }
W20260927 03:23:14.499684 5461 tablet.cc:2369] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:23:14.500090 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.499200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.0.98.1:38543)
0927 03:23:14.499254 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:14.499256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:14.499264 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:14.499307 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:14.499308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:14.499325 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:14.499374 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:14.499879 (+ 505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:14.499880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:14.499881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:14.499885 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:14.499901 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:14.499905 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:14.499951 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:14.500010 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 03:23:14.500021 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:23:14.501765 5414 raft_consensus.cc:1261] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0 [term 1 FOLLOWER]: Refusing update from remote peer f07637a65b53419db4d1b3b39b30e17f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:14.501775 5283 raft_consensus.cc:1261] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706 [term 1 FOLLOWER]: Refusing update from remote peer f07637a65b53419db4d1b3b39b30e17f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:14.502067 5481 consensus_queue.cc:1035] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b222b883cc4d7e9ae8a0d1a5644706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:14.502200 5484 consensus_queue.cc:1035] T 2c63fedc66ac404992d465cd6b7cd8e1 P f07637a65b53419db4d1b3b39b30e17f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d3246efd3cb48d5b0fcadf5ad1b6db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:14.504575 5489 mvcc.cc:205] Tried to move back new op lower bound from 7333803599876751360 to 7333803599781064704. Current Snapshot: MvccSnapshot[applied={T|T < 7333803599876751360}]
I20260927 03:23:14.506466 5009 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:14.506548 5009 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:14.507221 5009 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52e815688f2d46059f4afa4a32cea25c: Sending DeleteTablet for 3 replicas of tablet 2c63fedc66ac404992d465cd6b7cd8e1
I20260927 03:23:14.507583 5394 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c63fedc66ac404992d465cd6b7cd8e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:14 UTC) from {username='slave'} at 127.0.0.1:54466
I20260927 03:23:14.507649 5263 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c63fedc66ac404992d465cd6b7cd8e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:14 UTC) from {username='slave'} at 127.0.0.1:39300
I20260927 03:23:14.507858 5497 tablet_replica.cc:330] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706: stopping tablet replica
I20260927 03:23:14.507948 5497 raft_consensus.cc:2229] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:14.507961 5132 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c63fedc66ac404992d465cd6b7cd8e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:14 UTC) from {username='slave'} at 127.0.0.1:47030
I20260927 03:23:14.508057 5497 raft_consensus.cc:2258] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:14.508288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 5070
I20260927 03:23:14.508423 5497 ts_tablet_manager.cc:1905] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:14.509982 5497 ts_tablet_manager.cc:1918] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:14.510047 5497 log.cc:1196] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/2c63fedc66ac404992d465cd6b7cd8e1
I20260927 03:23:14.510035 5496 tablet_replica.cc:330] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0: stopping tablet replica
I20260927 03:23:14.510129 5496 raft_consensus.cc:2229] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:14.510231 5496 raft_consensus.cc:2258] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:14.510280 5497 ts_tablet_manager.cc:1939] T 2c63fedc66ac404992d465cd6b7cd8e1 P 39b222b883cc4d7e9ae8a0d1a5644706: Deleting consensus metadata
I20260927 03:23:14.510629 4994 catalog_manager.cc:4841] TS 39b222b883cc4d7e9ae8a0d1a5644706 (127.0.98.2:42069): tablet 2c63fedc66ac404992d465cd6b7cd8e1 (table test-table [id=a21a99617470435a83004fb65ed3272d]) successfully deleted
I20260927 03:23:14.510651 5496 ts_tablet_manager.cc:1905] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:14.512073 5496 ts_tablet_manager.cc:1918] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:14.512143 5496 log.cc:1196] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/2c63fedc66ac404992d465cd6b7cd8e1
I20260927 03:23:14.512359 5496 ts_tablet_manager.cc:1939] T 2c63fedc66ac404992d465cd6b7cd8e1 P 0d3246efd3cb48d5b0fcadf5ad1b6db0: Deleting consensus metadata
I20260927 03:23:14.512714 4995 catalog_manager.cc:4841] TS 0d3246efd3cb48d5b0fcadf5ad1b6db0 (127.0.98.3:42847): tablet 2c63fedc66ac404992d465cd6b7cd8e1 (table test-table [id=a21a99617470435a83004fb65ed3272d]) successfully deleted
W20260927 03:23:14.513710 4994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38543 (error 108)
I20260927 03:23:14.513900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 5201
I20260927 03:23:14.514474 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.514055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38543 (local address 127.0.0.1:47038)
0927 03:23:14.514401 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:14.514446 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327}
W20260927 03:23:14.514590 4994 catalog_manager.cc:4584] TS f07637a65b53419db4d1b3b39b30e17f (127.0.98.1:38543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c63fedc66ac404992d465cd6b7cd8e1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38543: connect: Connection refused (error 111)
I20260927 03:23:14.518062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 5332
I20260927 03:23:14.522253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 4979
2026-09-27T03:23:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (793 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 03:23:14.540238 392 mini_oidc.cc:152] Starting JWKS server
I20260927 03:23:14.541266 392 webserver.cc:460] Webserver started at https://127.0.0.1:33081/ using document root <none> and password file <none>
I20260927 03:23:14.541349 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 03:23:14.541491 392 webserver.cc:460] Webserver started at http://127.0.0.1:41491/ using document root <none> and password file <none>
2026-09-27T03:23:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:14Z Disabled control of system clock
I20260927 03:23:14.549147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40363
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40363
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33081/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/testchainca.cert with env {}
W20260927 03:23:14.628276 5505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:14.628494 5505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:14.628515 5505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:14.629647 5505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:14.629683 5505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:14.629693 5505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:14.629704 5505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:14.629714 5505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:23:14.629725 5505 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:23:14.629734 5505 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33081/jwks.json
I20260927 03:23:14.631232 5505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33081/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:14.631433 5505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:14.631595 5505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:14.636225 5513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:14.636351 5510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:14.636226 5511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:14.636430 5505 server_base.cc:1034] running on GCE node
I20260927 03:23:14.637580 5505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:14.637859 5505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:14.639002 5505 hybrid_clock.cc:648] HybridClock initialized: now 1790479394638982 us; error 36 us; skew 500 ppm
I20260927 03:23:14.640161 5505 webserver.cc:460] Webserver started at http://127.0.98.62:35855/ using document root <none> and password file <none>
I20260927 03:23:14.640379 5505 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:14.640429 5505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:14.640591 5505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:14.641669 5505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "88b041642843499a911587a2c8746589"
format_stamp: "Formatted at 2026-09-27 03:23:14 on dist-test-slave-gw2v"
server_key: "18ffe864600b8c86b52be2ef29f6eaca"
server_key_iv: "852204e503c8315c769c2ace6ad3d8e3"
server_key_version: "encryptionkey@0"
I20260927 03:23:14.642004 5505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "88b041642843499a911587a2c8746589"
format_stamp: "Formatted at 2026-09-27 03:23:14 on dist-test-slave-gw2v"
server_key: "18ffe864600b8c86b52be2ef29f6eaca"
server_key_iv: "852204e503c8315c769c2ace6ad3d8e3"
server_key_version: "encryptionkey@0"
I20260927 03:23:14.643311 5505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:14.644073 5519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:14.644219 5505 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:14.644294 5505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "88b041642843499a911587a2c8746589"
format_stamp: "Formatted at 2026-09-27 03:23:14 on dist-test-slave-gw2v"
server_key: "18ffe864600b8c86b52be2ef29f6eaca"
server_key_iv: "852204e503c8315c769c2ace6ad3d8e3"
server_key_version: "encryptionkey@0"
I20260927 03:23:14.644359 5505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:14.653582 5505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:14.653918 5505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:14.654052 5505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:14.657811 5505 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40363
I20260927 03:23:14.657845 5571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40363 every 8 connection(s)
I20260927 03:23:14.658177 5505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:14.659207 5572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:14.661640 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589: Bootstrap starting.
I20260927 03:23:14.662200 5572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:14.662437 5572 log.cc:824] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:14.663069 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589: No bootstrap required, opened a new log
I20260927 03:23:14.663503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 5505
I20260927 03:23:14.663581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:14.663725 392 external_mini_cluster.cc:1428] Setting key 32d5c24e4a21a6ac9f01c8c503dcc0e0
I20260927 03:23:14.664908 5572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88b041642843499a911587a2c8746589" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40363 } }
I20260927 03:23:14.665045 5572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:14.665078 5572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b041642843499a911587a2c8746589, State: Initialized, Role: FOLLOWER
I20260927 03:23:14.665220 5572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88b041642843499a911587a2c8746589" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40363 } }
I20260927 03:23:14.665304 5572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:14.665337 5572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:14.665403 5572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:14.665902 5572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88b041642843499a911587a2c8746589" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40363 } }
I20260927 03:23:14.665987 5572 leader_election.cc:302] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88b041642843499a911587a2c8746589; no voters:
I20260927 03:23:14.666138 5572 leader_election.cc:288] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:14.666182 5577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:14.666293 5577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [term 1 LEADER]: Becoming Leader. State: Replica: 88b041642843499a911587a2c8746589, State: Running, Role: LEADER
I20260927 03:23:14.666378 5577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88b041642843499a911587a2c8746589" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40363 } }
I20260927 03:23:14.666399 5572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:14.666759 5579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88b041642843499a911587a2c8746589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88b041642843499a911587a2c8746589" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40363 } } }
I20260927 03:23:14.666771 5578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88b041642843499a911587a2c8746589. Latest consensus state: current_term: 1 leader_uuid: "88b041642843499a911587a2c8746589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88b041642843499a911587a2c8746589" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40363 } } }
I20260927 03:23:14.666884 5579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:14.666893 5578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589 [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:14.667140 5583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:14.667670 5583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:14.669498 5583 catalog_manager.cc:1347] Generated new cluster ID: 3c5529f60ca1489283580625df1d1a89
I20260927 03:23:14.669548 5583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:23:14.670032 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.664215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49816 (local address 127.0.98.62:40363)
0927 03:23:14.664393 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:14.664400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:14.664426 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:23:14.664872 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:14.664878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:14.664932 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:14.665086 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:14.665300 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:14.666750 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:14.668037 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:14.668498 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:14.668616 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:14.669288 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:14.669291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:14.669292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:14.669489 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:14.669532 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:14.669547 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:14.669636 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:14.669714 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:23:14.669869 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260927 03:23:14.670881 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:23:14.679747 5583 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:14.680294 5583 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:14.683414 5583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88b041642843499a911587a2c8746589: Generated new TSK 0
I20260927 03:23:14.683600 5583 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 03:23:14.714284 5602 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 03:23:14.714496 5602 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 03:23:14.714500 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.691883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.0.98.62:40363)
0927 03:23:14.691957 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:14.691959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:14.692122 (+ 163us) server_negotiation.cc:407] Connection header received
0927 03:23:14.692227 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:14.692228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:14.692251 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:14.692305 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:23:14.692457 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:14.692924 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:14.693705 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:14.693895 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:14.693948 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:14.694144 (+ 196us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:23:14.714274 (+ 20130us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33081/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 03:23:14.714428 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49832: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33081/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":28}
W20260927 03:23:14.714561 5576 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49832: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33081/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 03:23:14.715197 5601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:14.691842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40363 (local address 127.0.0.1:49832)
0927 03:23:14.692053 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:14.692065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:14.692134 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:14.692318 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:14.692319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:14.692320 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:23:14.692397 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:14.692399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:14.693023 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:14.693024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:14.693626 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:14.693629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:14.693951 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:14.693954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:14.694017 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:14.694040 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:23:14.714772 (+ 20732us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33081/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 03:23:14.714936 (+ 164us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40363: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33081/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":110,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:40363: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33081/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 03:23:14.715754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 5505
2026-09-27T03:23:14Z chronyd exiting
I20260927 03:23:14.751030 392 test_util.cc:175] -----------------------------------------------
I20260927 03:23:14.751109 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479369225731-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (211 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 03:23:14.751819 392 mini_oidc.cc:152] Starting JWKS server
I20260927 03:23:14.752445 392 webserver.cc:460] Webserver started at https://127.0.0.1:45013/ using document root <none> and password file <none>
I20260927 03:23:14.752494 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 03:23:14.752602 392 webserver.cc:460] Webserver started at http://127.0.0.1:46089/ using document root <none> and password file <none>
2026-09-27T03:23:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:14Z Disabled control of system clock
I20260927 03:23:14.759632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34827
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34827
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45013/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/testchainca.cert with env {}
W20260927 03:23:14.838243 5608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:14.838474 5608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:14.838495 5608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:14.839707 5608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:14.839752 5608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:14.839764 5608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:14.839775 5608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:14.839787 5608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:23:14.839797 5608 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:23:14.839810 5608 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45013/jwks.json
I20260927 03:23:14.841343 5608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45013/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:14.841589 5608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:14.841791 5608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:14.846393 5613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:14.846482 5614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:14.846576 5616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:14.846688 5608 server_base.cc:1034] running on GCE node
I20260927 03:23:14.846944 5608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:14.847234 5608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:14.848377 5608 hybrid_clock.cc:648] HybridClock initialized: now 1790479394848364 us; error 30 us; skew 500 ppm
I20260927 03:23:14.849561 5608 webserver.cc:460] Webserver started at http://127.0.98.62:42175/ using document root <none> and password file <none>
I20260927 03:23:14.849767 5608 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:14.849814 5608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:14.849932 5608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:14.851011 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "7b91d4abd7d94b27a258a0b1f409cb9c"
format_stamp: "Formatted at 2026-09-27 03:23:14 on dist-test-slave-gw2v"
server_key: "5fb842b0efef8b10c2c7501d551e8c01"
server_key_iv: "633e03fb77806278493832cfab95e8c5"
server_key_version: "encryptionkey@0"
I20260927 03:23:14.851337 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b91d4abd7d94b27a258a0b1f409cb9c"
format_stamp: "Formatted at 2026-09-27 03:23:14 on dist-test-slave-gw2v"
server_key: "5fb842b0efef8b10c2c7501d551e8c01"
server_key_iv: "633e03fb77806278493832cfab95e8c5"
server_key_version: "encryptionkey@0"
I20260927 03:23:14.852530 5608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:14.853161 5622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:14.853327 5608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:14.853394 5608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "7b91d4abd7d94b27a258a0b1f409cb9c"
format_stamp: "Formatted at 2026-09-27 03:23:14 on dist-test-slave-gw2v"
server_key: "5fb842b0efef8b10c2c7501d551e8c01"
server_key_iv: "633e03fb77806278493832cfab95e8c5"
server_key_version: "encryptionkey@0"
I20260927 03:23:14.853469 5608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:14.867233 5608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:14.867534 5608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:14.867655 5608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:14.871340 5608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34827
I20260927 03:23:14.871382 5674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34827 every 8 connection(s)
I20260927 03:23:14.871698 5608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:14.872604 5675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:14.873641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 5608
I20260927 03:23:14.873723 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:14.873884 392 external_mini_cluster.cc:1428] Setting key 7592689ac5c5a13ae8ed7a377f34a62b
I20260927 03:23:14.875310 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c: Bootstrap starting.
I20260927 03:23:14.875939 5675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:14.876238 5675 log.cc:824] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:14.877018 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c: No bootstrap required, opened a new log
I20260927 03:23:14.879312 5678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.874358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41852 (local address 127.0.98.62:34827)
0927 03:23:14.874551 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:14.874558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:14.874583 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:23:14.875072 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:14.875080 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:14.875144 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:14.875263 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:14.875521 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:14.876474 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:14.877481 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:14.877870 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:14.877957 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:14.878680 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:14.878683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:14.878684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:14.878867 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:14.878902 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:14.878914 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:14.878963 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:14.879059 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 03:23:14.879122 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260927 03:23:14.879287 5675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b91d4abd7d94b27a258a0b1f409cb9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34827 } }
I20260927 03:23:14.879433 5675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:14.879487 5675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b91d4abd7d94b27a258a0b1f409cb9c, State: Initialized, Role: FOLLOWER
I20260927 03:23:14.879616 5675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b91d4abd7d94b27a258a0b1f409cb9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34827 } }
I20260927 03:23:14.879694 5675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:14.879719 5675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:14.879774 5675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:14.880200 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:23:14.880383 5675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b91d4abd7d94b27a258a0b1f409cb9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34827 } }
I20260927 03:23:14.880492 5675 leader_election.cc:302] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b91d4abd7d94b27a258a0b1f409cb9c; no voters:
I20260927 03:23:14.880661 5675 leader_election.cc:288] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:14.880684 5680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:14.880820 5680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [term 1 LEADER]: Becoming Leader. State: Replica: 7b91d4abd7d94b27a258a0b1f409cb9c, State: Running, Role: LEADER
I20260927 03:23:14.880906 5680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b91d4abd7d94b27a258a0b1f409cb9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34827 } }
I20260927 03:23:14.880913 5675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:14.881265 5680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b91d4abd7d94b27a258a0b1f409cb9c. Latest consensus state: current_term: 1 leader_uuid: "7b91d4abd7d94b27a258a0b1f409cb9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b91d4abd7d94b27a258a0b1f409cb9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34827 } } }
I20260927 03:23:14.881273 5681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b91d4abd7d94b27a258a0b1f409cb9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b91d4abd7d94b27a258a0b1f409cb9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34827 } } }
I20260927 03:23:14.881351 5680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:14.881354 5681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:14.881590 5685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:14.882126 5685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:14.883677 5685 catalog_manager.cc:1347] Generated new cluster ID: 43ec41c2172f40a88a3e62e2e8dd0f37
I20260927 03:23:14.883721 5685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:23:14.887470 5685 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:14.887980 5685 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:14.896224 5685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b91d4abd7d94b27a258a0b1f409cb9c: Generated new TSK 0
I20260927 03:23:14.896420 5685 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 03:23:14.917665 5705 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 03:23:14.917752 5705 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 03:23:14.918080 5678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:14.904271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.0.98.62:34827)
0927 03:23:14.904321 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:14.904323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:14.904432 (+ 109us) server_negotiation.cc:407] Connection header received
0927 03:23:14.904522 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:14.904522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:14.904534 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:14.904589 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:23:14.904719 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:14.905216 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:14.905989 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:14.906179 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:14.906231 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:14.906397 (+ 166us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:23:14.917851 (+ 11454us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 03:23:14.918010 (+ 159us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41866: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 03:23:14.918138 5704 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:14.904212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34827 (local address 127.0.0.1:41866)
0927 03:23:14.904366 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:14.904376 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:14.904446 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:14.904594 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:14.904595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:14.904597 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:23:14.904663 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:14.904665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:14.905310 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:14.905311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:14.905910 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:14.905912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:14.906240 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:14.906244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:14.906308 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:14.906313 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:23:14.917958 (+ 11645us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 03:23:14.918053 (+ 95us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34827: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
W20260927 03:23:14.918140 5678 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41866: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 03:23:14.918439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 5608
2026-09-27T03:23:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (203 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 03:23:14.955188 392 mini_oidc.cc:152] Starting JWKS server
I20260927 03:23:14.955838 392 webserver.cc:460] Webserver started at https://127.0.0.1:36963/ using document root <none> and password file <none>
I20260927 03:23:14.955883 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 03:23:14.956027 392 webserver.cc:460] Webserver started at http://127.0.0.1:33877/ using document root <none> and password file <none>
2026-09-27T03:23:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:14Z Disabled control of system clock
I20260927 03:23:14.963460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36805
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36805
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36963/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 03:23:15.044567 5711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:15.044790 5711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:15.044813 5711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:15.045944 5711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:15.045982 5711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:15.045995 5711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:15.046008 5711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:15.046020 5711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:23:15.046033 5711 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:23:15.046044 5711 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36963/jwks.json
I20260927 03:23:15.047461 5711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36963/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:15.047674 5711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:15.047919 5711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:15.052728 5719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:15.052795 5716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:15.052731 5717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:15.052919 5711 server_base.cc:1034] running on GCE node
I20260927 03:23:15.053228 5711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:15.053521 5711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:15.054670 5711 hybrid_clock.cc:648] HybridClock initialized: now 1790479395054650 us; error 34 us; skew 500 ppm
I20260927 03:23:15.055922 5711 webserver.cc:460] Webserver started at http://127.0.98.62:44571/ using document root <none> and password file <none>
I20260927 03:23:15.056152 5711 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:15.056207 5711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:15.056396 5711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:15.057557 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "a7af98aa05b643a3a1a73f85b54d4811"
format_stamp: "Formatted at 2026-09-27 03:23:15 on dist-test-slave-gw2v"
server_key: "3238771b8c45d50f9431eefbe3c6a2a9"
server_key_iv: "95a748675e21e721e823d03dfc9fbb8d"
server_key_version: "encryptionkey@0"
I20260927 03:23:15.057905 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "a7af98aa05b643a3a1a73f85b54d4811"
format_stamp: "Formatted at 2026-09-27 03:23:15 on dist-test-slave-gw2v"
server_key: "3238771b8c45d50f9431eefbe3c6a2a9"
server_key_iv: "95a748675e21e721e823d03dfc9fbb8d"
server_key_version: "encryptionkey@0"
I20260927 03:23:15.059326 5711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:23:15.060142 5725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:15.060388 5711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:15.060477 5711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "a7af98aa05b643a3a1a73f85b54d4811"
format_stamp: "Formatted at 2026-09-27 03:23:15 on dist-test-slave-gw2v"
server_key: "3238771b8c45d50f9431eefbe3c6a2a9"
server_key_iv: "95a748675e21e721e823d03dfc9fbb8d"
server_key_version: "encryptionkey@0"
I20260927 03:23:15.060540 5711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:15.093853 5711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:15.094170 5711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:15.094295 5711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:15.098593 5711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36805
I20260927 03:23:15.098635 5777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36805 every 8 connection(s)
I20260927 03:23:15.099027 5711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:15.099732 5778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:15.102295 5778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811: Bootstrap starting.
I20260927 03:23:15.102871 5778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:15.103125 5778 log.cc:824] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:15.103807 5778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811: No bootstrap required, opened a new log
I20260927 03:23:15.105326 5778 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7af98aa05b643a3a1a73f85b54d4811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36805 } }
I20260927 03:23:15.105520 5778 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:15.105543 5778 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7af98aa05b643a3a1a73f85b54d4811, State: Initialized, Role: FOLLOWER
I20260927 03:23:15.105671 5778 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7af98aa05b643a3a1a73f85b54d4811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36805 } }
I20260927 03:23:15.105732 5778 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:15.105751 5778 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:15.105777 5778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:15.106247 5778 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7af98aa05b643a3a1a73f85b54d4811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36805 } }
I20260927 03:23:15.106331 5778 leader_election.cc:302] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7af98aa05b643a3a1a73f85b54d4811; no voters:
I20260927 03:23:15.106475 5778 leader_election.cc:288] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:15.106530 5781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:15.106699 5781 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [term 1 LEADER]: Becoming Leader. State: Replica: a7af98aa05b643a3a1a73f85b54d4811, State: Running, Role: LEADER
I20260927 03:23:15.106730 5778 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:15.106797 5781 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7af98aa05b643a3a1a73f85b54d4811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36805 } }
I20260927 03:23:15.107188 5783 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7af98aa05b643a3a1a73f85b54d4811. Latest consensus state: current_term: 1 leader_uuid: "a7af98aa05b643a3a1a73f85b54d4811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7af98aa05b643a3a1a73f85b54d4811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36805 } } }
I20260927 03:23:15.107277 5783 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:15.107304 5782 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7af98aa05b643a3a1a73f85b54d4811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7af98aa05b643a3a1a73f85b54d4811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36805 } } }
I20260927 03:23:15.107362 5782 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811 [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:15.107467 5790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:15.107815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 5711
I20260927 03:23:15.107908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:15.107970 5790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:15.108070 392 external_mini_cluster.cc:1428] Setting key 18125d31a66fff25be1bc4d1c9ec8883
I20260927 03:23:15.109551 5790 catalog_manager.cc:1347] Generated new cluster ID: c9373711ea3f418eb70d928ce98464b9
I20260927 03:23:15.109611 5790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:23:15.114089 5790 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:15.114327 5800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:15.108515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.0.98.62:36805)
0927 03:23:15.109033 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:15.109040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:15.109067 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:23:15.109183 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:15.109190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:15.109271 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:15.109410 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:15.109639 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:15.111026 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:15.112077 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:15.112736 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:15.112857 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:15.113505 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:15.113509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:15.113509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:15.113714 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:15.113757 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:15.113771 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:15.113869 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:15.113994 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 03:23:15.114071 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":66,"threads_started":1}
I20260927 03:23:15.114722 5790 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:15.115437 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:23:15.127524 5790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7af98aa05b643a3a1a73f85b54d4811: Generated new TSK 0
I20260927 03:23:15.127730 5790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:23:15.142488 5800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:15.138765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.0.98.62:36805)
0927 03:23:15.138849 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:15.138851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:15.138968 (+ 117us) server_negotiation.cc:407] Connection header received
0927 03:23:15.139032 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:15.139033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:15.139043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:15.139116 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:23:15.139242 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:15.139741 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:15.140885 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:15.141079 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:15.141129 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:15.141317 (+ 188us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:23:15.142271 (+ 954us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36963/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 03:23:15.142407 (+ 136us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36963/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":31}
W20260927 03:23:15.142560 5800 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36963/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 03:23:15.142627 5807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:15.138690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36805 (local address 127.0.0.1:49450)
0927 03:23:15.138884 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:15.138896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:15.138963 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:15.139117 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:15.139118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:15.139120 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:23:15.139185 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:15.139188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:15.139814 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:15.139815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:15.140798 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:15.140801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:15.141153 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:15.141157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:15.141217 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:15.141224 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:23:15.142369 (+ 1145us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36963/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 03:23:15.142486 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36805: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36963/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
I20260927 03:23:15.142920 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 5711
E20260927 03:23:15.153327 5808 webserver.cc:533] Webserver: sslize failed:
E20260927 03:23:15.153400 5808 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T03:23:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (224 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T03:23:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:15Z Disabled control of system clock
I20260927 03:23:15.187173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46375
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46375
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/testchainca.cert with env {}
W20260927 03:23:15.265826 5812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:15.266041 5812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:15.266062 5812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:15.267210 5812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:15.267242 5812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:15.267253 5812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:15.267264 5812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:15.267279 5812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:23:15.267290 5812 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:23:15.267303 5812 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 03:23:15.268725 5812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46375
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:15.268931 5812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:15.269093 5812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:15.273829 5820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:15.273829 5817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:15.273903 5818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:15.273901 5812 server_base.cc:1034] running on GCE node
I20260927 03:23:15.274318 5812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:15.274638 5812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:15.275784 5812 hybrid_clock.cc:648] HybridClock initialized: now 1790479395275758 us; error 41 us; skew 500 ppm
I20260927 03:23:15.276938 5812 webserver.cc:460] Webserver started at http://127.0.98.62:44767/ using document root <none> and password file <none>
I20260927 03:23:15.277161 5812 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:15.277209 5812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:15.277333 5812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:15.278421 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "e4b48982900144b0aef77a82800c7aee"
format_stamp: "Formatted at 2026-09-27 03:23:15 on dist-test-slave-gw2v"
server_key: "292568a6feb33e36115109f5541a639c"
server_key_iv: "cfba598732651f0ecd6c6ecb657c7f11"
server_key_version: "encryptionkey@0"
I20260927 03:23:15.278748 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4b48982900144b0aef77a82800c7aee"
format_stamp: "Formatted at 2026-09-27 03:23:15 on dist-test-slave-gw2v"
server_key: "292568a6feb33e36115109f5541a639c"
server_key_iv: "cfba598732651f0ecd6c6ecb657c7f11"
server_key_version: "encryptionkey@0"
I20260927 03:23:15.280023 5812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:15.280717 5826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:15.280917 5812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:23:15.281006 5812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "e4b48982900144b0aef77a82800c7aee"
format_stamp: "Formatted at 2026-09-27 03:23:15 on dist-test-slave-gw2v"
server_key: "292568a6feb33e36115109f5541a639c"
server_key_iv: "cfba598732651f0ecd6c6ecb657c7f11"
server_key_version: "encryptionkey@0"
I20260927 03:23:15.281077 5812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:15.293952 5812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:15.294266 5812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:15.294399 5812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:15.298085 5812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46375
I20260927 03:23:15.298138 5878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46375 every 8 connection(s)
I20260927 03:23:15.298451 5812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:15.299412 5879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:15.301198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 5812
I20260927 03:23:15.301292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:15.301478 392 external_mini_cluster.cc:1428] Setting key 030f428cd499141c3b7b23df7e3049b6
I20260927 03:23:15.301888 5879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee: Bootstrap starting.
I20260927 03:23:15.302487 5879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:15.302739 5879 log.cc:824] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:15.303356 5879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee: No bootstrap required, opened a new log
I20260927 03:23:15.305073 5879 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b48982900144b0aef77a82800c7aee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46375 } }
I20260927 03:23:15.305207 5879 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:15.305236 5879 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b48982900144b0aef77a82800c7aee, State: Initialized, Role: FOLLOWER
I20260927 03:23:15.305361 5879 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b48982900144b0aef77a82800c7aee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46375 } }
I20260927 03:23:15.305431 5879 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:15.305469 5879 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:15.305505 5879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:15.306010 5879 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b48982900144b0aef77a82800c7aee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46375 } }
I20260927 03:23:15.306103 5879 leader_election.cc:302] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4b48982900144b0aef77a82800c7aee; no voters:
I20260927 03:23:15.306250 5879 leader_election.cc:288] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:15.306285 5884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:15.306382 5884 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [term 1 LEADER]: Becoming Leader. State: Replica: e4b48982900144b0aef77a82800c7aee, State: Running, Role: LEADER
I20260927 03:23:15.306469 5884 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b48982900144b0aef77a82800c7aee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46375 } }
I20260927 03:23:15.306509 5879 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:15.306766 5884 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4b48982900144b0aef77a82800c7aee. Latest consensus state: current_term: 1 leader_uuid: "e4b48982900144b0aef77a82800c7aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b48982900144b0aef77a82800c7aee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46375 } } }
I20260927 03:23:15.306778 5885 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4b48982900144b0aef77a82800c7aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b48982900144b0aef77a82800c7aee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46375 } } }
I20260927 03:23:15.307022 5885 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:15.307099 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:15.302037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.0.98.62:46375)
0927 03:23:15.302224 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:15.302231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:15.302264 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:23:15.302748 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:15.302756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:15.302858 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:15.303005 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:15.303224 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:15.304159 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:15.304922 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:15.305361 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:15.305472 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:15.306093 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:15.306097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:15.306097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:15.306292 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:15.306336 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:15.306351 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:15.306659 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:15.306690 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 03:23:15.306779 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260927 03:23:15.307603 5893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:15.306866 5884 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:15.308028 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:23:15.308082 5893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:15.309502 5893 catalog_manager.cc:1347] Generated new cluster ID: 5a0847e45654413aa47f1ba2c19c845b
I20260927 03:23:15.309547 5893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:23:15.322914 5893 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:15.323448 5893 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:15.330916 5893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4b48982900144b0aef77a82800c7aee: Generated new TSK 0
I20260927 03:23:15.331128 5893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:23:15.778079 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:15.338571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.0.98.62:46375)
0927 03:23:15.338632 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:15.338634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:15.338734 (+ 100us) server_negotiation.cc:407] Connection header received
0927 03:23:15.338839 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:15.338840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:15.338852 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:15.338927 (+ 75us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:23:15.339060 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:15.339552 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:15.340664 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:15.340853 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:15.340894 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:15.341056 (+ 162us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:23:15.777845 (+436789us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 03:23:15.778018 (+ 173us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43892: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 03:23:15.778146 5882 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43892: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 03:23:15.778321 5908 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:15.338501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46375 (local address 127.0.0.1:43892)
0927 03:23:15.338666 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:15.338678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:15.338751 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:15.338932 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:15.338933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:15.338935 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:23:15.339003 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:15.339005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:15.339619 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:15.339620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:15.340585 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:15.340588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:15.340909 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:15.340913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:15.340972 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:15.340977 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:23:15.778000 (+437023us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 03:23:15.778137 (+ 137us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46375: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260927 03:23:15.778685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 5812
2026-09-27T03:23:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (615 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:23:16 dist-test-slave-gw2v krb5kdc[5920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:23:16 dist-test-slave-gw2v krb5kdc[5920](info): set up 2 sockets
Sep 27 03:23:16 dist-test-slave-gw2v krb5kdc[5920](info): commencing operation
krb5kdc: starting...
W20260927 03:23:16.910884 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:23:16 dist-test-slave-gw2v krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:23:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:23:17.025477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36599
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:17.105060 5933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:17.105300 5933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:17.105324 5933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:17.106544 5933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:17.106586 5933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:17.106599 5933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:17.106622 5933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:17.106644 5933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:17.108248 5933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:17.108495 5933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:17.108708 5933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:17.113394 5939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:17.113404 5938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:17.113412 5941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:17.113448 5933 server_base.cc:1034] running on GCE node
I20260927 03:23:17.113803 5933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:17.114063 5933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:17.115226 5933 hybrid_clock.cc:648] HybridClock initialized: now 1790479397115184 us; error 27 us; skew 500 ppm
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:17.122113 5933 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:23:17.122598 5933 webserver.cc:460] Webserver started at http://127.0.98.62:35789/ using document root <none> and password file <none>
I20260927 03:23:17.122793 5933 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:17.122862 5933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:17.123003 5933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:17.124048 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "4b9f4a9cdd0448a284b5acf82657e12f"
format_stamp: "Formatted at 2026-09-27 03:23:17 on dist-test-slave-gw2v"
server_key: "b2634707313dc27733f060deec0a1a47"
server_key_iv: "1e46131f3fd6bb70df544af1a42b5579"
server_key_version: "encryptionkey@0"
I20260927 03:23:17.124377 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b9f4a9cdd0448a284b5acf82657e12f"
format_stamp: "Formatted at 2026-09-27 03:23:17 on dist-test-slave-gw2v"
server_key: "b2634707313dc27733f060deec0a1a47"
server_key_iv: "1e46131f3fd6bb70df544af1a42b5579"
server_key_version: "encryptionkey@0"
I20260927 03:23:17.125614 5933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:23:17.126245 5948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:17.126389 5933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:23:17.126466 5933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "4b9f4a9cdd0448a284b5acf82657e12f"
format_stamp: "Formatted at 2026-09-27 03:23:17 on dist-test-slave-gw2v"
server_key: "b2634707313dc27733f060deec0a1a47"
server_key_iv: "1e46131f3fd6bb70df544af1a42b5579"
server_key_version: "encryptionkey@0"
I20260927 03:23:17.126528 5933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:17.137462 5933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:17.139673 5933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:17.139804 5933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:17.143584 5933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36599
I20260927 03:23:17.143626 6000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36599 every 8 connection(s)
I20260927 03:23:17.144090 5933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:17.144685 6001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:17.146876 6001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f: Bootstrap starting.
I20260927 03:23:17.147415 6001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:17.147636 6001 log.cc:824] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:17.148237 6001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f: No bootstrap required, opened a new log
I20260927 03:23:17.149777 6001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f4a9cdd0448a284b5acf82657e12f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36599 } }
I20260927 03:23:17.149917 6001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:17.149938 6001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9f4a9cdd0448a284b5acf82657e12f, State: Initialized, Role: FOLLOWER
I20260927 03:23:17.150050 6001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f4a9cdd0448a284b5acf82657e12f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36599 } }
I20260927 03:23:17.150115 6001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:17.150152 6001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:17.150198 6001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:17.150292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 5933
I20260927 03:23:17.150382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:17.150558 392 external_mini_cluster.cc:1428] Setting key 98496d2d1b17e85d19da4af4c620306d
I20260927 03:23:17.151305 6001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f4a9cdd0448a284b5acf82657e12f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36599 } }
I20260927 03:23:17.151403 6001 leader_election.cc:302] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b9f4a9cdd0448a284b5acf82657e12f; no voters:
I20260927 03:23:17.151546 6001 leader_election.cc:288] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:17.151585 6006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:17.151744 6006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [term 1 LEADER]: Becoming Leader. State: Replica: 4b9f4a9cdd0448a284b5acf82657e12f, State: Running, Role: LEADER
I20260927 03:23:17.151859 6001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:17.151875 6006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f4a9cdd0448a284b5acf82657e12f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36599 } }
I20260927 03:23:17.152211 6007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b9f4a9cdd0448a284b5acf82657e12f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f4a9cdd0448a284b5acf82657e12f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36599 } } }
I20260927 03:23:17.152227 6008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b9f4a9cdd0448a284b5acf82657e12f. Latest consensus state: current_term: 1 leader_uuid: "4b9f4a9cdd0448a284b5acf82657e12f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9f4a9cdd0448a284b5acf82657e12f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36599 } } }
I20260927 03:23:17.152307 6007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:17.152325 6008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:17.153101 6014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:17.153532 6014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:17.154794 6014 catalog_manager.cc:1347] Generated new cluster ID: 969c6a7e47544d979083eb293fec16c9
I20260927 03:23:17.154863 6014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:17.161722 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:17.151037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.0.98.62:36599)
0927 03:23:17.151214 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:17.151226 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:17.151275 (+ 49us) server_negotiation.cc:407] Connection header received
0927 03:23:17.151382 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:17.151388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:17.151445 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:17.151571 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:17.152309 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.153669 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.154704 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.155320 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.155410 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:17.158194 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:17.158199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:17.158199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:17.158211 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:17.160112 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:17.160465 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:17.160468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:17.160468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:17.160581 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:17.160785 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:17.160786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:17.160786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:17.161093 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:17.161245 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:17.161373 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 03:23:17.161508 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260927 03:23:17.162163 6014 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:17.162858 6014 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:17.167320 6014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b9f4a9cdd0448a284b5acf82657e12f: Generated new TSK 0
I20260927 03:23:17.167495 6014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:23:17.195341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36599
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:17.275166 6027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:17.275380 6027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:17.275411 6027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:17.276620 6027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:17.276671 6027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:17.276714 6027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:23:17.278245 6027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:17.278491 6027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:17.278707 6027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:17.283530 6032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:17.283519 6033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:17.283520 6035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:17.283552 6027 server_base.cc:1034] running on GCE node
I20260927 03:23:17.283901 6027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:17.284116 6027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:17.285254 6027 hybrid_clock.cc:648] HybridClock initialized: now 1790479397285237 us; error 34 us; skew 500 ppm
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:17.287688 6027 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:23:17.288105 6027 webserver.cc:460] Webserver started at http://127.0.98.1:44391/ using document root <none> and password file <none>
I20260927 03:23:17.288311 6027 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:17.288359 6027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:17.288487 6027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:17.289695 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "d224cce2e8b74e46a159fad89fd03d3f"
format_stamp: "Formatted at 2026-09-27 03:23:17 on dist-test-slave-gw2v"
server_key: "88497f7db6b64d7d32f2f31a91cfe031"
server_key_iv: "e075380c1672a353692ab1e798119957"
server_key_version: "encryptionkey@0"
I20260927 03:23:17.290107 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d224cce2e8b74e46a159fad89fd03d3f"
format_stamp: "Formatted at 2026-09-27 03:23:17 on dist-test-slave-gw2v"
server_key: "88497f7db6b64d7d32f2f31a91cfe031"
server_key_iv: "e075380c1672a353692ab1e798119957"
server_key_version: "encryptionkey@0"
I20260927 03:23:17.291487 6027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:17.292356 6042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:17.292532 6027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:17.292613 6027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "d224cce2e8b74e46a159fad89fd03d3f"
format_stamp: "Formatted at 2026-09-27 03:23:17 on dist-test-slave-gw2v"
server_key: "88497f7db6b64d7d32f2f31a91cfe031"
server_key_iv: "e075380c1672a353692ab1e798119957"
server_key_version: "encryptionkey@0"
I20260927 03:23:17.292671 6027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:17.321412 6027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:17.323705 6027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:17.323848 6027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:17.324075 6027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:17.324406 6027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:17.324441 6027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:17.324471 6027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:17.324488 6027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:17.330044 6027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46445
I20260927 03:23:17.330123 6155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46445 every 8 connection(s)
I20260927 03:23:17.330456 6027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:17.339123 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:17.331145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53291 (local address 127.0.98.62:36599)
0927 03:23:17.331221 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:17.331224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:17.331412 (+ 188us) server_negotiation.cc:407] Connection header received
0927 03:23:17.332316 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:17.332317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:17.332327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:17.332377 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:17.332953 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.333452 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.334433 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.334624 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.334673 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:17.336728 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:17.336729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:17.336729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:17.336737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:17.337835 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:17.338175 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:17.338175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:17.338175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:17.338220 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:17.338419 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:17.338419 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:17.338420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:17.338517 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:17.338564 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:17.338993 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 03:23:17.339055 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 03:23:17.339397 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:17.331119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36599 (local address 127.0.98.1:53291)
0927 03:23:17.331292 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:17.331331 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:17.332229 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:17.332436 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:17.332439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:17.332688 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:17.332890 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:17.332894 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.333535 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.333536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:17.334363 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:17.334366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.334685 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.334685 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:17.334785 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:17.335312 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:17.335321 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:17.336636 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:17.337905 (+ 1269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:17.337911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:17.337912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:17.338121 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:17.338278 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:17.338278 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:17.338279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:17.338374 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:17.338677 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:17.338683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:17.338846 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 03:23:17.338965 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 03:23:17.339156 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260927 03:23:17.339958 6156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36599
I20260927 03:23:17.340045 6156 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:17.340191 6156 heartbeater.cc:511] Master 127.0.98.62:36599 requested a full tablet report, sending...
I20260927 03:23:17.340514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 6027
I20260927 03:23:17.340608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:23:17.340745 5965 ts_manager.cc:194] Registered new tserver with Master: d224cce2e8b74e46a159fad89fd03d3f (127.0.98.1:46445)
I20260927 03:23:17.340771 392 external_mini_cluster.cc:1428] Setting key a26355579c9c675718d8d930bbe5ca1b
I20260927 03:23:17.341629 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53291
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:23:17.365483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36599
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:17.444948 6161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:17.445169 6161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:17.445199 6161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:17.446343 6161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:17.446386 6161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:17.446420 6161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:23:17.447892 6161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:17.448127 6161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:17.448338 6161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:17.453343 6169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:17.453349 6167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:17.453343 6166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:17.453622 6161 server_base.cc:1034] running on GCE node
I20260927 03:23:17.453827 6161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:17.454048 6161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:17.455220 6161 hybrid_clock.cc:648] HybridClock initialized: now 1790479397455205 us; error 28 us; skew 500 ppm
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:17.457801 6161 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:23:17.458374 6161 webserver.cc:460] Webserver started at http://127.0.98.2:45041/ using document root <none> and password file <none>
I20260927 03:23:17.458575 6161 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:17.458632 6161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:17.458756 6161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:17.459844 6161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "d675185d283a4cf49fcba563f3cd6ca6"
format_stamp: "Formatted at 2026-09-27 03:23:17 on dist-test-slave-gw2v"
server_key: "26d59c9dd841cdec825da3db1d0775f5"
server_key_iv: "88e505f78be9e6ef426d00d2128f7cef"
server_key_version: "encryptionkey@0"
I20260927 03:23:17.460177 6161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d675185d283a4cf49fcba563f3cd6ca6"
format_stamp: "Formatted at 2026-09-27 03:23:17 on dist-test-slave-gw2v"
server_key: "26d59c9dd841cdec825da3db1d0775f5"
server_key_iv: "88e505f78be9e6ef426d00d2128f7cef"
server_key_version: "encryptionkey@0"
I20260927 03:23:17.461380 6161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:23:17.461995 6176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:17.462215 6161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:17.462301 6161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "d675185d283a4cf49fcba563f3cd6ca6"
format_stamp: "Formatted at 2026-09-27 03:23:17 on dist-test-slave-gw2v"
server_key: "26d59c9dd841cdec825da3db1d0775f5"
server_key_iv: "88e505f78be9e6ef426d00d2128f7cef"
server_key_version: "encryptionkey@0"
I20260927 03:23:17.462363 6161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:17.482481 6161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:17.484687 6161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:17.484827 6161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:17.485064 6161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:17.485379 6161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:17.485414 6161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:17.485442 6161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:17.485460 6161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:17.490948 6161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33643
I20260927 03:23:17.491010 6289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33643 every 8 connection(s)
I20260927 03:23:17.491334 6161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:17.500247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 6161
I20260927 03:23:17.500339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:23:17.500512 392 external_mini_cluster.cc:1428] Setting key 0cffb6b7f26be7c6a87789f1372d5fdf
I20260927 03:23:17.500999 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:17.492080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50947 (local address 127.0.98.62:36599)
0927 03:23:17.492177 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:17.492179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:17.492365 (+ 186us) server_negotiation.cc:407] Connection header received
0927 03:23:17.493260 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:17.493261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:17.493272 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:17.493331 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:17.493982 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.494495 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.495653 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.495869 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.495931 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:17.498039 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:17.498041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:17.498041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:17.498053 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:17.499542 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:17.499923 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:17.499924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:17.499925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:17.499990 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:17.500274 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:17.500275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:17.500276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:17.500418 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:17.500476 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:17.500803 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 03:23:17.500915 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 03:23:17.501163 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:17.492057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36599 (local address 127.0.98.2:50947)
0927 03:23:17.492222 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:17.492265 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:17.493168 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:17.493404 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:17.493407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:17.493675 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:17.493878 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:17.493882 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.494600 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.494600 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:17.495563 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:17.495566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.495936 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.495937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:17.496039 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:17.496565 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:17.496574 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:17.497925 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:17.499616 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:17.499621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:17.499623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:17.499842 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:17.500056 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:17.500056 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:17.500057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:17.500191 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:17.500477 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:17.500483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:17.500653 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 03:23:17.500765 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 03:23:17.500956 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260927 03:23:17.501559 6290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36599
I20260927 03:23:17.501644 6290 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:17.501814 6290 heartbeater.cc:511] Master 127.0.98.62:36599 requested a full tablet report, sending...
I20260927 03:23:17.502153 5965 ts_manager.cc:194] Registered new tserver with Master: d675185d283a4cf49fcba563f3cd6ca6 (127.0.98.2:33643)
I20260927 03:23:17.502713 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50947
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:23:17.523536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36599
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:17.602792 6295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:17.603013 6295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:17.603052 6295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:17.604174 6295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:17.604212 6295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:17.604246 6295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:23:17.605679 6295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:17.605904 6295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:17.606120 6295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:17.610870 6303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:17.610880 6301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:17.611073 6295 server_base.cc:1034] running on GCE node
W20260927 03:23:17.610877 6300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:17.611410 6295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:17.611650 6295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:17.612789 6295 hybrid_clock.cc:648] HybridClock initialized: now 1790479397612778 us; error 30 us; skew 500 ppm
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:17.615221 6295 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:23:17.615620 6295 webserver.cc:460] Webserver started at http://127.0.98.3:44019/ using document root <none> and password file <none>
I20260927 03:23:17.615823 6295 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:17.615870 6295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:17.615990 6295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:17.617035 6295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "9e130d4da19c4954949ebadac2bb8717"
format_stamp: "Formatted at 2026-09-27 03:23:17 on dist-test-slave-gw2v"
server_key: "de6a0789b79cf75a2dd6aac90604537f"
server_key_iv: "ae631a7e914637b61d96c67b32a738ce"
server_key_version: "encryptionkey@0"
I20260927 03:23:17.617360 6295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9e130d4da19c4954949ebadac2bb8717"
format_stamp: "Formatted at 2026-09-27 03:23:17 on dist-test-slave-gw2v"
server_key: "de6a0789b79cf75a2dd6aac90604537f"
server_key_iv: "ae631a7e914637b61d96c67b32a738ce"
server_key_version: "encryptionkey@0"
I20260927 03:23:17.618547 6295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:23:17.619344 6310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:17.619503 6295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:23:17.619585 6295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "9e130d4da19c4954949ebadac2bb8717"
format_stamp: "Formatted at 2026-09-27 03:23:17 on dist-test-slave-gw2v"
server_key: "de6a0789b79cf75a2dd6aac90604537f"
server_key_iv: "ae631a7e914637b61d96c67b32a738ce"
server_key_version: "encryptionkey@0"
I20260927 03:23:17.619652 6295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:17.651137 6295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:17.653402 6295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:17.653533 6295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:17.653750 6295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:17.654031 6295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:17.654064 6295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:17.654093 6295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:17.654127 6295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:17.659565 6295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44161
I20260927 03:23:17.659613 6423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44161 every 8 connection(s)
I20260927 03:23:17.659914 6295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:17.668200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 6295
I20260927 03:23:17.668282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:23:17.668447 392 external_mini_cluster.cc:1428] Setting key f4402da39db6dd7007fc80e32c2e7955
I20260927 03:23:17.670042 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:17.660523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41869 (local address 127.0.98.62:36599)
0927 03:23:17.660570 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:17.660572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:17.660809 (+ 237us) server_negotiation.cc:407] Connection header received
0927 03:23:17.662046 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:17.662048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:17.662061 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:17.662105 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:17.662730 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.663285 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.664484 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.664795 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.664861 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:17.667413 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:17.667415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:17.667415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:17.667421 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:17.668913 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:17.669325 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:17.669326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:17.669326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:17.669387 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:17.669558 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:17.669558 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:17.669558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:17.669686 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:17.669734 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:17.669923 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 03:23:17.669980 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:23:17.670255 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:17.660515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36599 (local address 127.0.98.3:41869)
0927 03:23:17.660678 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:17.660725 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:17.661934 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:17.662186 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:17.662190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:17.662442 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:17.662636 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:17.662640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.663368 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.663370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:17.664382 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:17.664386 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.664881 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.664882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:17.665070 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:17.665653 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:17.665663 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:17.667322 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:17.668987 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:17.668995 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:17.668997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:17.669274 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:17.669429 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:17.669429 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:17.669430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:17.669518 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:17.669723 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:17.669725 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:17.669837 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 03:23:17.669910 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 03:23:17.670027 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260927 03:23:17.670626 6424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36599
I20260927 03:23:17.670717 6424 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:17.670899 6424 heartbeater.cc:511] Master 127.0.98.62:36599 requested a full tablet report, sending...
I20260927 03:23:17.671226 5965 ts_manager.cc:194] Registered new tserver with Master: 9e130d4da19c4954949ebadac2bb8717 (127.0.98.3:44161)
I20260927 03:23:17.671550 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41869
I20260927 03:23:17.680446 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/test-admin.keytab.
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:17.694577 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 03:23:17.696159 6431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:17.695472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.0.98.1:46445)
0927 03:23:17.695631 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:17.695635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:17.695644 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:23:17.695670 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:17.695671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:17.695685 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:17.695762 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:17.696100 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44646: BlockingRecv error: recv got EOF from 127.0.0.1:44646 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
W20260927 03:23:17.696271 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:17.697165 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:17.697929 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:23:17.704896 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:17.698430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.0.98.1:46445)
0927 03:23:17.698476 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:17.698477 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:17.698503 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:23:17.698553 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:17.698554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:17.698563 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:17.698606 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:17.698967 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.699689 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.700382 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.700711 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.700766 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:17.702628 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:17.702634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:17.702635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:17.702641 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:17.704044 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:17.704311 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:17.704314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:17.704314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:17.704362 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:17.704488 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:17.704488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:17.704489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:17.704600 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:17.704703 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:17.704739 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 03:23:17.704794 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260927 03:23:17.706544 6431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:17.705995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.0.98.1:46445)
0927 03:23:17.706031 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:17.706033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:17.706085 (+ 52us) server_negotiation.cc:407] Connection header received
0927 03:23:17.706149 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:17.706150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:17.706160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:17.706224 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:17.706503 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44662: BlockingRecv error: recv got EOF from 127.0.0.1:44662 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:23:17.706612 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:17.707473 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:17.708202 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:23:17.714228 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:17.708696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.0.98.1:46445)
0927 03:23:17.708733 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:17.708735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:17.708797 (+ 62us) server_negotiation.cc:407] Connection header received
0927 03:23:17.708842 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:17.708842 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:17.708851 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:17.708894 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:17.709227 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.709689 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.710338 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.710524 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.710572 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:17.712312 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:17.712313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:17.712314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:17.712320 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:17.713470 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:17.713719 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:17.713720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:17.713720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:17.713761 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:17.713874 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:17.713875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:17.713875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:17.713947 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:17.714006 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:17.714073 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 03:23:17.714125 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 03:23:17.715818 6431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:17.715328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.0.98.1:46445)
0927 03:23:17.715386 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:17.715387 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:17.715396 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:23:17.715460 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:17.715460 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:17.715469 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:17.715526 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:17.715776 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44684: BlockingRecv error: recv got EOF from 127.0.0.1:44684 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 03:23:17.715869 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479369225731-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:17.716892 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:17.717759 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:23:17.724041 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:17.718240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.0.98.1:46445)
0927 03:23:17.718301 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:17.718302 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:17.718310 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:17.718357 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:17.718358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:17.718366 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:17.718410 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:17.718741 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.719265 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.719923 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:17.720120 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:17.720170 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:17.721886 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:17.721888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:17.721888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:17.721895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:17.723155 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:17.723417 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:17.723418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:17.723418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:17.723468 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:17.723616 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:17.723617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:17.723617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:17.723709 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:17.723787 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:17.723881 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 03:23:17.723953 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:23:17.724552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 6027
I20260927 03:23:17.728885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 6161
I20260927 03:23:17.732268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 6295
I20260927 03:23:17.735824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 5933
2026-09-27T03:23:17Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1784 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[6444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[6444](info): set up 2 sockets
Sep 27 03:23:17 dist-test-slave-gw2v krb5kdc[6444](info): commencing operation
krb5kdc: starting...
W20260927 03:23:18.678854 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:23:18 dist-test-slave-gw2v krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:23:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 03:23:18.789727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37939
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:18.869678 6457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:18.869911 6457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:18.869935 6457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:18.871217 6457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:18.871265 6457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:18.871279 6457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:18.871291 6457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:18.871302 6457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:18.872931 6457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:18.873184 6457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:18.873405 6457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:18.878352 6462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:18.878357 6465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:18.878361 6463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:18.878604 6457 server_base.cc:1034] running on GCE node
I20260927 03:23:18.878888 6457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:18.879243 6457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:18.880385 6457 hybrid_clock.cc:648] HybridClock initialized: now 1790479398880364 us; error 39 us; skew 500 ppm
Sep 27 03:23:18 dist-test-slave-gw2v krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479398, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:18.882792 6457 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 03:23:18.883253 6457 webserver.cc:460] Webserver started at http://127.0.98.62:38645/ using document root <none> and password file <none>
I20260927 03:23:18.883440 6457 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:18.883482 6457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:18.883580 6457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:18.884598 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "4216573a82c74be9b92dbe0e0046c1ba"
format_stamp: "Formatted at 2026-09-27 03:23:18 on dist-test-slave-gw2v"
server_key: "87b18207075ca77626c47571662eb043"
server_key_iv: "c7abb4cfc8077821eb89596df4b71c6a"
server_key_version: "encryptionkey@0"
I20260927 03:23:18.884907 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "4216573a82c74be9b92dbe0e0046c1ba"
format_stamp: "Formatted at 2026-09-27 03:23:18 on dist-test-slave-gw2v"
server_key: "87b18207075ca77626c47571662eb043"
server_key_iv: "c7abb4cfc8077821eb89596df4b71c6a"
server_key_version: "encryptionkey@0"
I20260927 03:23:18.886168 6457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:23:18.886868 6472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:18.887007 6457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:18.887130 6457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "4216573a82c74be9b92dbe0e0046c1ba"
format_stamp: "Formatted at 2026-09-27 03:23:18 on dist-test-slave-gw2v"
server_key: "87b18207075ca77626c47571662eb043"
server_key_iv: "c7abb4cfc8077821eb89596df4b71c6a"
server_key_version: "encryptionkey@0"
I20260927 03:23:18.887207 6457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:18.909260 6457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:18.911257 6457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:18.911368 6457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:18.915010 6457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37939
I20260927 03:23:18.915077 6524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37939 every 8 connection(s)
I20260927 03:23:18.915393 6457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:18.916321 6525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:18.918588 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba: Bootstrap starting.
I20260927 03:23:18.919185 6525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:18.919445 6525 log.cc:824] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:18.920126 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba: No bootstrap required, opened a new log
I20260927 03:23:18.921660 6525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } }
I20260927 03:23:18.921800 6525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:18.921823 6525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4216573a82c74be9b92dbe0e0046c1ba, State: Initialized, Role: FOLLOWER
I20260927 03:23:18.921968 6525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } }
I20260927 03:23:18.922042 6525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:18.922075 6525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:18.922122 6525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:18.922938 6525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } }
I20260927 03:23:18.923040 6525 leader_election.cc:302] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4216573a82c74be9b92dbe0e0046c1ba; no voters:
I20260927 03:23:18.923197 6525 leader_election.cc:288] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:18.923231 6528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:18.923354 6528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [term 1 LEADER]: Becoming Leader. State: Replica: 4216573a82c74be9b92dbe0e0046c1ba, State: Running, Role: LEADER
I20260927 03:23:18.923458 6525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:18.923449 6528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } }
I20260927 03:23:18.923848 6529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } }
I20260927 03:23:18.923944 6529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:18.923862 6530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4216573a82c74be9b92dbe0e0046c1ba. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } }
I20260927 03:23:18.924082 6530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:18.924192 6535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:18.924677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 6457
I20260927 03:23:18.924693 6535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:18.924768 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:18.924938 392 external_mini_cluster.cc:1428] Setting key ad9ba82d2d768d5c0cee5f5b4c049a69
I20260927 03:23:18.926192 6535 catalog_manager.cc:1347] Generated new cluster ID: 5476a318fb324d70a21b536467118dfc
I20260927 03:23:18.926246 6535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:23:18 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:23:18.935052 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:18.925798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44920 (local address 127.0.98.62:37939)
0927 03:23:18.926006 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:18.926015 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:18.926047 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:23:18.926919 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:18.926926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:18.926975 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:18.927111 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:18.927714 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:18.928632 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:18.929376 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:18.929819 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:18.929918 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:18.932082 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:18.932086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:18.932086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:18.932098 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:18.933598 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:18.933900 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:18.933903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:18.933904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:18.933996 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:18.934164 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:18.934164 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:18.934165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:18.934403 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:18.934544 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:18.934644 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 03:23:18.934777 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":166,"thread_start_us":61,"threads_started":1}
I20260927 03:23:18.936836 6535 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:18.937536 6535 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:18.945863 6535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba: Generated new TSK 0
I20260927 03:23:18.946090 6535 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 03:23:18.975914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37939
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:19.091838 6551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:19.092116 6551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:19.092150 6551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:19.093817 6551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:19.093874 6551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:19.093909 6551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:23:19.095853 6551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:19.096112 6551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:19.096364 6551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:19.101549 6557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:19.101557 6556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:19.101605 6551 server_base.cc:1034] running on GCE node
W20260927 03:23:19.101577 6559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:19.101974 6551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:19.102221 6551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:19.103356 6551 hybrid_clock.cc:648] HybridClock initialized: now 1790479399103344 us; error 36 us; skew 500 ppm
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479399, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:19.105594 6551 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 03:23:19.106007 6551 webserver.cc:460] Webserver started at http://127.0.98.1:35921/ using document root <none> and password file <none>
I20260927 03:23:19.106204 6551 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:19.106256 6551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:19.106374 6551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:19.107470 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "387121f928344d86a04295c84ae74dd7"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "570830e5ba11302dd502edf173a00c29"
server_key_iv: "4b1d1ffd8f574190c02ce561aa81b807"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.107795 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "387121f928344d86a04295c84ae74dd7"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "570830e5ba11302dd502edf173a00c29"
server_key_iv: "4b1d1ffd8f574190c02ce561aa81b807"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.109022 6551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:23:19.109750 6566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.109915 6551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:19.109992 6551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "387121f928344d86a04295c84ae74dd7"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "570830e5ba11302dd502edf173a00c29"
server_key_iv: "4b1d1ffd8f574190c02ce561aa81b807"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.110050 6551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:19.135277 6551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:19.137166 6551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:19.137285 6551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:19.137490 6551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:19.137780 6551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:19.137814 6551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.137843 6551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:19.137867 6551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.143224 6551 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42473
I20260927 03:23:19.143263 6679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42473 every 8 connection(s)
I20260927 03:23:19.143558 6551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479399, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:23:19.151764 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.144208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48315 (local address 127.0.98.62:37939)
0927 03:23:19.144271 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.144273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.144441 (+ 168us) server_negotiation.cc:407] Connection header received
0927 03:23:19.144894 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.144895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.144907 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.144973 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.145826 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.146318 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.147309 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.147505 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.147555 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.149393 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.149394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.149395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.149402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.150652 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.151035 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.151035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.151036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.151087 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.151257 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.151258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.151258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.151349 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.151389 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.151604 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.151697 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:23:19.151842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 6551
I20260927 03:23:19.151925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:23:19.151923 6682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.144178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.1:48315)
0927 03:23:19.144318 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.144360 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.144822 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.145035 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.145038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.145566 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.145755 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.145759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.146388 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.146389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.147238 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.147241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.147563 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.147563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.147653 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.148064 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.148072 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.149299 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.150724 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.150730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.150731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.150975 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.151134 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.151134 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.151135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.151223 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.151387 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.151390 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.151499 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.151574 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.151702 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260927 03:23:19.152118 392 external_mini_cluster.cc:1428] Setting key 7d221acf903b1a07ff28c7db598a2603
I20260927 03:23:19.152628 6680 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37939
I20260927 03:23:19.152726 6680 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:19.152894 6680 heartbeater.cc:511] Master 127.0.98.62:37939 requested a full tablet report, sending...
I20260927 03:23:19.153420 6489 ts_manager.cc:194] Registered new tserver with Master: 387121f928344d86a04295c84ae74dd7 (127.0.98.1:42473)
I20260927 03:23:19.154086 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48315
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 03:23:19.175457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37939
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:19.254877 6685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:19.255110 6685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:19.255133 6685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:19.256311 6685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:19.256351 6685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:19.256381 6685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:23:19.257800 6685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:19.258026 6685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:19.258236 6685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:19.262977 6693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:19.263031 6691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:19.262995 6690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:19.263024 6685 server_base.cc:1034] running on GCE node
I20260927 03:23:19.263394 6685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:19.263644 6685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:19.264775 6685 hybrid_clock.cc:648] HybridClock initialized: now 1790479399264766 us; error 29 us; skew 500 ppm
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479399, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:19.267062 6685 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 03:23:19.267488 6685 webserver.cc:460] Webserver started at http://127.0.98.2:36647/ using document root <none> and password file <none>
I20260927 03:23:19.267695 6685 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:19.267742 6685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:19.267860 6685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:19.268889 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "ef4d9fd028fb4283b48cc865333cb3ca"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "c1440a728ac92779a650d0bf819be425"
server_key_iv: "70af8e60acbf7cc750557d96fc40cb84"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.269222 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ef4d9fd028fb4283b48cc865333cb3ca"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "c1440a728ac92779a650d0bf819be425"
server_key_iv: "70af8e60acbf7cc750557d96fc40cb84"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.270449 6685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:19.271139 6700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.271274 6685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.271353 6685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "ef4d9fd028fb4283b48cc865333cb3ca"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "c1440a728ac92779a650d0bf819be425"
server_key_iv: "70af8e60acbf7cc750557d96fc40cb84"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.271411 6685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:19.293423 6685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:19.295207 6685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:19.295342 6685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:19.295555 6685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:19.295889 6685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:19.295928 6685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.295957 6685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:19.295975 6685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.301383 6685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36333
I20260927 03:23:19.301436 6813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36333 every 8 connection(s)
I20260927 03:23:19.301730 6685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479399, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:23:19.310267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 6685
I20260927 03:23:19.310372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:23:19.310592 392 external_mini_cluster.cc:1428] Setting key eb6e2058a0e30d538c7afa95abb1ce0f
I20260927 03:23:19.310597 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.302400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51105 (local address 127.0.98.62:37939)
0927 03:23:19.302476 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.302478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.302676 (+ 198us) server_negotiation.cc:407] Connection header received
0927 03:23:19.303258 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.303258 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.303268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.303313 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.304194 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.304682 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.305673 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.305854 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.305896 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.307995 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.307997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.307997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.308004 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.309142 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.309476 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.309477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.309477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.309522 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.309826 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.309827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.309827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.309955 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.310021 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.310447 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.310525 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 03:23:19.310779 6816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.302369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.2:51105)
0927 03:23:19.302549 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.302588 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.303175 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.303380 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.303384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.303938 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.304129 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.304134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.304752 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.304753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.305612 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.305614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.305952 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.305953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.306060 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.306498 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.306506 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.307878 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.309214 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.309219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.309222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.309425 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.309595 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.309596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.309596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.309723 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.310042 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.310050 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.310230 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.310368 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.310549 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
I20260927 03:23:19.311259 6814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37939
I20260927 03:23:19.311359 6814 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:19.311524 6814 heartbeater.cc:511] Master 127.0.98.62:37939 requested a full tablet report, sending...
I20260927 03:23:19.311867 6489 ts_manager.cc:194] Registered new tserver with Master: ef4d9fd028fb4283b48cc865333cb3ca (127.0.98.2:36333)
I20260927 03:23:19.312217 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51105
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 03:23:19.333233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37939
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:19.411654 6819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:19.411855 6819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:19.411886 6819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:19.413118 6819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:19.413164 6819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:19.413199 6819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:23:19.414705 6819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:19.414992 6819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:19.415223 6819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:19.420240 6825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:19.420241 6824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:19.420385 6827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:19.420486 6819 server_base.cc:1034] running on GCE node
I20260927 03:23:19.420663 6819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:19.420897 6819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:19.422034 6819 hybrid_clock.cc:648] HybridClock initialized: now 1790479399422015 us; error 36 us; skew 500 ppm
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479399, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:19.424584 6819 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 03:23:19.424997 6819 webserver.cc:460] Webserver started at http://127.0.98.3:46669/ using document root <none> and password file <none>
I20260927 03:23:19.425207 6819 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:19.425254 6819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:19.425372 6819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:19.426440 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "bbd404a350704e0483c7b2f08f07d77e"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "fc7a4dbd24d188808990e3251799f815"
server_key_iv: "7d5a4cf550e218cc2f577ee856d0bfca"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.426766 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bbd404a350704e0483c7b2f08f07d77e"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "fc7a4dbd24d188808990e3251799f815"
server_key_iv: "7d5a4cf550e218cc2f577ee856d0bfca"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.428014 6819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:19.428632 6834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.428763 6819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:23:19.428839 6819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "bbd404a350704e0483c7b2f08f07d77e"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "fc7a4dbd24d188808990e3251799f815"
server_key_iv: "7d5a4cf550e218cc2f577ee856d0bfca"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.428895 6819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:19.443429 6819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:19.445565 6819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:19.445716 6819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:19.445943 6819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:19.446225 6819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:19.446257 6819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.446286 6819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:19.446302 6819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.451705 6819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45035
I20260927 03:23:19.451751 6947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45035 every 8 connection(s)
I20260927 03:23:19.452052 6819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:23:19.457502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 6819
I20260927 03:23:19.457614 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:23:19.457777 392 external_mini_cluster.cc:1428] Setting key d65067970efba2aaa3bac90f3db3d23f
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479399, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:23:19.460639 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.452707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54997 (local address 127.0.98.62:37939)
0927 03:23:19.452781 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.452783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.452989 (+ 206us) server_negotiation.cc:407] Connection header received
0927 03:23:19.453448 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.453449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.453460 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.453508 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.454415 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.454968 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.455920 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.456106 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.456148 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.458266 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.458266 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.458267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.458274 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.459657 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.459986 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.459987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.459987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.460032 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.460190 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.460190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.460191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.460282 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.460322 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.460526 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.460579 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:23:19.460814 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.452685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.3:54997)
0927 03:23:19.452852 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.452909 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.453365 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.453574 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.453577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.454150 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.454343 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.454347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.455042 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.455043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.455859 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.455861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.456158 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.456159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.456250 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.456715 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.456724 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.458193 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.459719 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.459724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.459726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.459928 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.460070 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.460070 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.460071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.460154 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.460321 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.460324 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.460432 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.460506 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.460634 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260927 03:23:19.461254 6948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37939
I20260927 03:23:19.461334 6948 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:19.461483 6948 heartbeater.cc:511] Master 127.0.98.62:37939 requested a full tablet report, sending...
I20260927 03:23:19.461794 6489 ts_manager.cc:194] Registered new tserver with Master: bbd404a350704e0483c7b2f08f07d77e (127.0.98.3:45035)
I20260927 03:23:19.462154 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54997
I20260927 03:23:19.469552 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 03:23:19.493491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33793
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37939,127.0.98.61:33793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:19.572211 6953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:19.572434 6953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:19.572464 6953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:19.573652 6953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:19.573700 6953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:19.573719 6953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:19.573733 6953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:19.573748 6953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:19.575296 6953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37939,127.0.98.61:33793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
W20260927 03:23:19.575420 6953 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:37939,127.0.98.61:33793'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 03:23:19.575584 6953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:19.575790 6953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:19.580571 6961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:19.580617 6953 server_base.cc:1034] running on GCE node
W20260927 03:23:19.580583 6958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:19.580586 6959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:19.581106 6953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:19.581408 6953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:19.582578 6953 hybrid_clock.cc:648] HybridClock initialized: now 1790479399582564 us; error 29 us; skew 500 ppm
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479399, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:19.584929 6953 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 03:23:19.585386 6953 webserver.cc:460] Webserver started at http://127.0.98.61:37799/ using document root <none> and password file <none>
I20260927 03:23:19.585577 6953 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:19.585624 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:19.585743 6953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:19.586753 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/data/instance:
uuid: "af86687a976b42eeb8255db2ef798344"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "14e4c3d1cf61e69c5f6b863bfe8fd21e"
server_key_iv: "deb0cea34bd537e19dc065ff063ef6da"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.587136 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/wal/instance:
uuid: "af86687a976b42eeb8255db2ef798344"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "14e4c3d1cf61e69c5f6b863bfe8fd21e"
server_key_iv: "deb0cea34bd537e19dc065ff063ef6da"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.588330 6953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:19.588971 6968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.589116 6953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:19.589195 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/wal
uuid: "af86687a976b42eeb8255db2ef798344"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "14e4c3d1cf61e69c5f6b863bfe8fd21e"
server_key_iv: "deb0cea34bd537e19dc065ff063ef6da"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.589264 6953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:19.604390 6953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:19.606298 6953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:19.606428 6953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:19.610397 6953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33793
I20260927 03:23:19.610433 7020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33793 every 8 connection(s)
I20260927 03:23:19.610761 6953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/data/info.pb
I20260927 03:23:19.611691 7021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:19.613209 7021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:23:19.618165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 6953
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479399, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:23:19.621536 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.613883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.0.98.62:37939)
0927 03:23:19.613945 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.613947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.614109 (+ 162us) server_negotiation.cc:407] Connection header received
0927 03:23:19.614550 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.614551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.614560 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.614617 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.615228 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.615751 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.616734 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.616946 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.617003 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.619079 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.619080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.619080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.619087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.620314 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.620659 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.620660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.620660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.620711 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.620916 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.620916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.620917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.621058 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.621114 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.621361 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.621439 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:23:19.621891 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.613847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:43776)
0927 03:23:19.613987 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.614028 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.614473 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.614681 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.614684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.614957 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.615150 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.615154 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.615821 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.615822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.616647 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.616648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.617008 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.617009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.617104 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.617541 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.617549 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.619001 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.620387 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.620393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.620395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.620597 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.620763 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.620764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.620764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.620864 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.621115 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.621120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.621250 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.621335 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.621476 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260927 03:23:19.622524 7021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 03:23:19.626221 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.618617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.0.98.61:33793)
0927 03:23:19.618761 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.618765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.618928 (+ 163us) server_negotiation.cc:407] Connection header received
0927 03:23:19.618971 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.618974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.618996 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.619058 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.619700 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.620348 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.621041 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.621334 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.621391 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.623696 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.623703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.623703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.623710 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.625115 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.625429 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.625432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.625432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.625517 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.625680 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.625680 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.625681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.625828 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.625992 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.626047 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.626119 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479399, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 03:23:19.629201 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.622747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33793 (local address 127.0.0.1:55850)
0927 03:23:19.622787 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.622796 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.622880 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.623057 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.623058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.623181 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.623221 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.623223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.623877 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.623878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.624691 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.624693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.625068 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.625068 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.625110 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.625423 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.625427 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.626634 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.628518 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.628518 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.628519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.628669 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.628802 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.628802 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.628803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.628840 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.628982 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.628983 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.629013 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.629059 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.629125 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 03:23:19.629290 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.622804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55850 (local address 127.0.98.61:33793)
0927 03:23:19.622971 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.622972 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.622977 (+ 5us) server_negotiation.cc:407] Connection header received
0927 03:23:19.622990 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.622990 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.622997 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.623049 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.623276 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.623792 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.624758 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.624979 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.625034 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.626706 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.626707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.626707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.626713 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.628441 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.628725 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.628725 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.628725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.628765 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.628870 (+ 105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.628870 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.628870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.628948 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.628991 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.629073 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.629165 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":150,"thread_start_us":113,"threads_started":1}
I20260927 03:23:19.630373 7021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: Bootstrap starting.
I20260927 03:23:19.630981 7021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:19.631217 7021 log.cc:824] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:19.631888 7021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: No bootstrap required, opened a new log
I20260927 03:23:19.633234 7021 raft_consensus.cc:348] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } }
I20260927 03:23:19.633353 7021 raft_consensus.cc:374] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:19.633385 7021 raft_consensus.cc:729] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af86687a976b42eeb8255db2ef798344, State: Initialized, Role: FOLLOWER
I20260927 03:23:19.633507 7021 consensus_queue.cc:260] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } }
I20260927 03:23:19.633774 7021 sys_catalog.cc:564] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:19.633737 7028 sys_catalog.cc:455] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } } }
I20260927 03:23:19.633823 7028 sys_catalog.cc:458] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:23:19.635228 7039 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:23:19.635272 7039 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:23:19.636673 6953 master_runner.cc:428] Detected that this master af86687a976b42eeb8255db2ef798344 is joining an existing cluster
I20260927 03:23:19.636718 6953 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:33793
I20260927 03:23:19.643981 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.637708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.0.98.62:37939)
0927 03:23:19.637753 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.637755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.637868 (+ 113us) server_negotiation.cc:407] Connection header received
0927 03:23:19.637960 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.637961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.637972 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.638019 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.638501 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.639065 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.640074 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.640309 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.640340 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.641415 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.641417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.641418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.641425 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.642933 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.643225 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.643226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.643227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.643277 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.643431 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.643432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.643432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.643543 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.643603 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.643821 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.643890 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:23:19.644405 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.637670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:43784)
0927 03:23:19.637806 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.637816 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.637891 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.638063 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.638064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.638318 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.638412 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.638414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.639132 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.639133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.639982 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.639984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.640412 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.640413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.640453 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.640892 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.640896 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.641319 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.642997 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.642998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.642999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.643166 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.643327 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.643328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.643328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.643382 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.643627 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.643628 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.643681 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.644209 (+ 528us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.644308 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":3,"spinlock_wait_cycles":10624,"thread_start_us":116,"threads_started":1}
I20260927 03:23:19.644508 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.637828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.0.98.61:33793)
0927 03:23:19.637862 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.637864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.637937 (+ 73us) server_negotiation.cc:407] Connection header received
0927 03:23:19.638002 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.638003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.638012 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.638053 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.638482 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.639251 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.640162 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.640464 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.640533 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.641590 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.641591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.641591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.641598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.643485 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.643806 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.643807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.643808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.643861 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.644032 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.644033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.644033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.644149 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.644206 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.644354 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.644425 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:23:19.644733 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.637672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33793 (local address 127.0.0.1:55856)
0927 03:23:19.637883 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.637892 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.637956 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.638064 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.638065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.638313 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.638416 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.638417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.639333 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.639334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.640095 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.640097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.640542 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.640543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.640595 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.641121 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.641125 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.641512 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.643551 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.643552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.643552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.643746 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.643920 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.643921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.643921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.643975 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.644228 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.644229 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.644263 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.644572 (+ 309us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.644650 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":14,"thread_start_us":89,"threads_started":1}
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479398, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 03:23:19.653219 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.646423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33793 (local address 127.0.0.1:55870)
0927 03:23:19.646589 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.646627 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.646685 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.646814 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.646837 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.647092 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.647142 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.647144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.648024 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.648025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.648591 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.648592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.649017 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.649017 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.649075 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.649471 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.649476 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.650747 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.652298 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.652305 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.652307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.652496 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.652676 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.652677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.652677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.652726 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.652951 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.652955 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.652998 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.653056 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.653147 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260927 03:23:19.653256 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.646476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.0.98.61:33793)
0927 03:23:19.646516 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.646518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.646683 (+ 165us) server_negotiation.cc:407] Connection header received
0927 03:23:19.646738 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.646739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.646748 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.646810 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.647203 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.647950 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.648654 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.648945 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.649008 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.650860 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.650861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.650862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.650869 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.652230 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.652569 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.652570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.652570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.652624 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.652772 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.652773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.652773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.652898 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.652970 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.653092 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.653164 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:23:19.654191 6489 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:33793: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "af86687a976b42eeb8255db2ef798344"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 33793
}
attrs {
promote: true
}
}
dest_uuid: "4216573a82c74be9b92dbe0e0046c1ba"
cas_config_opid_index: -1
I20260927 03:23:19.654497 6489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: true } }
I20260927 03:23:19.654966 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER af86687a976b42eeb8255db2ef798344 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: true } } }
I20260927 03:23:19.655238 6995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:19.656018 7048 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:33793
I20260927 03:23:19.656083 6995 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 FOLLOWER]: Refusing update from remote peer 4216573a82c74be9b92dbe0e0046c1ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 03:23:19.656242 6953 master.cc:537] Master@127.0.98.61:33793 shutting down...
I20260927 03:23:19.656446 7048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:19.656860 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: true } } }
I20260927 03:23:19.656944 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: This master's current role is: LEADER
W20260927 03:23:19.657114 6476 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 03:23:19.657790 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba -> Peer af86687a976b42eeb8255db2ef798344 (127.0.98.61:33793): Couldn't send request to peer af86687a976b42eeb8255db2ef798344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 03:23:19.659188 6953 raft_consensus.cc:2229] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:19.659279 6953 raft_consensus.cc:2258] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:19.659325 6953 tablet_replica.cc:330] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: stopping tablet replica
I20260927 03:23:19.665779 6953 master.cc:552] Master@127.0.98.61:33793 shutdown complete.
I20260927 03:23:19.665831 6953 master_runner.cc:305] Clearing existing system tablet
I20260927 03:23:19.665961 6953 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:19.666004 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:19.667003 6953 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:19.667582 7054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.667850 6953 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:19.668120 6953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:23:19.668188 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/wal
uuid: "af86687a976b42eeb8255db2ef798344"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "14e4c3d1cf61e69c5f6b863bfe8fd21e"
server_key_iv: "deb0cea34bd537e19dc065ff063ef6da"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.669015 6953 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:19.670557 6953 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 03:23:19.670624 6953 log.cc:1196] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 03:23:19.670862 6953 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: Deleting consensus metadata
I20260927 03:23:19.670981 6953 master_runner.cc:315] Copying system tablet from 127.0.98.62:37939
I20260927 03:23:19.671574 6953 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37939
I20260927 03:23:19.677947 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.671949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.0.98.62:37939)
0927 03:23:19.671994 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.671997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.672123 (+ 126us) server_negotiation.cc:407] Connection header received
0927 03:23:19.672188 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.672189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.672198 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.672240 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.672570 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.673048 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.674093 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.674331 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.674381 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.675688 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.675689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.675689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.675696 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.676915 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.677320 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.677320 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.677321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.677373 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.677539 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.677540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.677540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.677634 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.677676 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.677828 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.677883 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:23:19.678027 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.671857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:43794)
0927 03:23:19.672040 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.672052 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.672127 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.672258 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.672259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.672452 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.672512 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.672514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.673113 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.673114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.674020 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.674022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.674385 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.674386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.674454 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.675212 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.675217 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.675612 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.677007 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.677008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.677009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.677221 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.677429 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.677429 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.677429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.677491 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.677681 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.677682 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.677729 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.677812 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.677913 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":71,"threads_started":1}
I20260927 03:23:19.678334 6508 tablet_copy_service.cc:140] P 4216573a82c74be9b92dbe0e0046c1ba: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer af86687a976b42eeb8255db2ef798344 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43794)
I20260927 03:23:19.678438 6508 tablet_copy_service.cc:161] P 4216573a82c74be9b92dbe0e0046c1ba: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer af86687a976b42eeb8255db2ef798344 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43794: session id = af86687a976b42eeb8255db2ef798344-00000000000000000000000000000000
I20260927 03:23:19.679303 6508 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 03:23:19.679970 6953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 03:23:19.681208 6953 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: tablet copy: Starting download of 0 data blocks...
I20260927 03:23:19.681360 6953 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: tablet copy: Starting download of 1 WAL segments...
I20260927 03:23:19.682221 6953 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 03:23:19.683187 6508 tablet_copy_service.cc:342] P 4216573a82c74be9b92dbe0e0046c1ba: Request end of tablet copy session af86687a976b42eeb8255db2ef798344-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43794
I20260927 03:23:19.683246 6508 tablet_copy_service.cc:433] P 4216573a82c74be9b92dbe0e0046c1ba: ending tablet copy session af86687a976b42eeb8255db2ef798344-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer af86687a976b42eeb8255db2ef798344
I20260927 03:23:19.685622 6953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:19.686776 7061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:19.686976 7062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:19.687115 6953 server_base.cc:1034] running on GCE node
W20260927 03:23:19.690938 7064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:19.691083 6953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:19.691226 6953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:19.692337 6953 hybrid_clock.cc:648] HybridClock initialized: now 1790479399692332 us; error 26 us; skew 500 ppm
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479399, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:19.693552 6953 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 03:23:19.693792 6953 webserver.cc:460] Webserver started at http://127.0.98.61:34033/ using document root <none> and password file <none>
I20260927 03:23:19.693902 6953 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:19.693936 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:19.694487 6953 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.694993 7071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.695111 6953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.695152 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/wal
uuid: "af86687a976b42eeb8255db2ef798344"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "14e4c3d1cf61e69c5f6b863bfe8fd21e"
server_key_iv: "deb0cea34bd537e19dc065ff063ef6da"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.695209 6953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:19.706190 6953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:19.707712 6953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:19.710474 6953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33793
I20260927 03:23:19.710646 7123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33793 every 8 connection(s)
I20260927 03:23:19.710788 6953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-1/data/info.pb
I20260927 03:23:19.711447 7124 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:23:19.711830 7124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: Bootstrap starting.
I20260927 03:23:19.713944 7124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:23:19.714129 7124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: Bootstrap complete.
I20260927 03:23:19.714298 7124 raft_consensus.cc:348] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: true } }
I20260927 03:23:19.714362 7124 raft_consensus.cc:729] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: af86687a976b42eeb8255db2ef798344, State: Initialized, Role: LEARNER
I20260927 03:23:19.714388 7124 consensus_queue.cc:260] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: true } }
I20260927 03:23:19.714550 7124 sys_catalog.cc:564] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:19.714560 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: true } } }
I20260927 03:23:19.714666 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:23:19.715884 7138 catalog_manager.cc:1259] Loaded cluster ID: 5476a318fb324d70a21b536467118dfc
I20260927 03:23:19.715931 7138 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: loading cluster ID for follower catalog manager: success
I20260927 03:23:19.716974 7138 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: acquiring CA information for follower catalog manager: success
I20260927 03:23:19.717231 7138 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:23:19.734620 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.727688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55896 (local address 127.0.98.61:33793)
0927 03:23:19.727812 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.727814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.727823 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:23:19.727850 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.727851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.727862 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.727953 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.728441 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.729469 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.730185 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.730493 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.730569 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.731775 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.731777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.731777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.731784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.733319 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.733639 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.733640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.733640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.733719 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.733879 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.733880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.733880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.734067 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.734181 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.734250 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.734329 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 03:23:19.757827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33087
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37939,127.0.98.61:33793,127.0.98.60:33087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:19.837858 7142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:19.838081 7142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:19.838111 7142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:19.839295 7142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:19.839339 7142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:19.839356 7142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:19.839372 7142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:19.839388 7142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:19.840898 7142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37939,127.0.98.61:33793,127.0.98.60:33087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:19.841140 7142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:19.841341 7142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:19.845986 7150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:19.846002 7148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:19.845983 7147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:19.846187 7142 server_base.cc:1034] running on GCE node
I20260927 03:23:19.846495 7142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:19.846771 7142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:19.847905 7142 hybrid_clock.cc:648] HybridClock initialized: now 1790479399847886 us; error 35 us; skew 500 ppm
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479399, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:19.850147 7142 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 03:23:19.850560 7142 webserver.cc:460] Webserver started at http://127.0.98.60:46681/ using document root <none> and password file <none>
I20260927 03:23:19.850745 7142 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:19.850785 7142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:19.850916 7142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:19.851926 7142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/data/instance:
uuid: "2c20d31f9fb64676a11df189db754b54"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "cae6d33c8efef16d0421bf01ba331ac4"
server_key_iv: "1d5078462d3932d4e901fa91258bfba9"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.852248 7142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/wal/instance:
uuid: "2c20d31f9fb64676a11df189db754b54"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "cae6d33c8efef16d0421bf01ba331ac4"
server_key_iv: "1d5078462d3932d4e901fa91258bfba9"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.853487 7142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:23:19.854239 7157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.854414 7142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:19.854493 7142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/wal
uuid: "2c20d31f9fb64676a11df189db754b54"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "cae6d33c8efef16d0421bf01ba331ac4"
server_key_iv: "1d5078462d3932d4e901fa91258bfba9"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.854579 7142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:19.869290 7142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:19.871289 7142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:19.871404 7142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:19.875554 7142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33087
I20260927 03:23:19.875599 7209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33087 every 8 connection(s)
I20260927 03:23:19.875960 7142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/data/info.pb
I20260927 03:23:19.876886 7210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:19.878350 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:23:19.882761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 7142
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479399, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:23:19.887844 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.879183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.0.98.62:37939)
0927 03:23:19.879243 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.879245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.879326 (+ 81us) server_negotiation.cc:407] Connection header received
0927 03:23:19.879798 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.879800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.879813 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.879864 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.880443 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.880963 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.882360 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.882579 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.882633 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.885019 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.885021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.885021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.885030 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.886319 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.886738 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.886739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.886740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.886795 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.887179 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.887180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.887180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.887296 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.887357 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.887667 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.887748 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:23:19.887995 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.879056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:43802)
0927 03:23:19.879205 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.879251 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.879705 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.879932 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.879936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.880174 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.880356 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.880360 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.881043 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.881044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.882273 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.882275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.882648 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.882649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.882769 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.883339 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.883350 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.884877 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.886388 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.886392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.886394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.886663 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.886952 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.886953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.886953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.887089 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.887359 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.887364 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.887528 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.887647 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.887795 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 03:23:19.888665 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:23:19.892043 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.883304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.0.98.60:33087)
0927 03:23:19.883475 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.883480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.883500 (+ 20us) server_negotiation.cc:407] Connection header received
0927 03:23:19.883521 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.883523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.883544 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.883627 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.884340 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.885232 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.885926 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.886222 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.886280 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.888740 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.888745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.888745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.888751 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.890514 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.890961 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.890965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.890965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.891029 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.891220 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.891221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.891221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.891385 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.891498 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.891808 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.891881 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479399, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 03:23:19.895339 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.888912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33793 (local address 127.0.0.1:55912)
0927 03:23:19.888956 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.888967 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.889005 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.889164 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.889166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.889313 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.889358 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.889359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.890208 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.890209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.890803 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.890806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.891275 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.891276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.891331 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.891651 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.891656 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.893009 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.894397 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.894397 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.894398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.894546 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.894714 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.894715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.894716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.894774 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.895103 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.895104 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.895150 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.895200 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.895274 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 03:23:19.895344 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.889020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55912 (local address 127.0.98.61:33793)
0927 03:23:19.889069 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.889070 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.889078 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:19.889099 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.889100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.889113 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.889158 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.889423 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.890130 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.890914 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.891207 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.891270 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.893090 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.893091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.893091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.893098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.894321 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.894603 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.894604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.894604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.894646 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.894860 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.894860 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.894861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.895035 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.895109 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.895219 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.895281 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:23:19.895758 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479399, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 03:23:19.901321 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.895992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33087 (local address 127.0.0.1:45742)
0927 03:23:19.896031 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.896041 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.896112 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.896212 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.896213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.896313 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.896343 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.896344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.896875 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.896876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.897400 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.897401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.897705 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.897706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.897749 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.898126 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.898130 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.899256 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.900553 (+ 1297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.900554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.900554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.900697 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.900858 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.900859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.900859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.900902 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.901094 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.901094 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.901125 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.901170 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.901240 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 03:23:19.901371 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.896020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45742 (local address 127.0.98.60:33087)
0927 03:23:19.896050 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.896051 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.896075 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:23:19.896167 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.896167 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.896176 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.896210 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.896380 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.896824 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.897463 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.897639 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.897680 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.899335 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.899336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.899337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.899342 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.900484 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.900753 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.900753 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.900753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.900797 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.900952 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.900953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.900953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.901038 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.901080 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.901187 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.901280 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":15}
I20260927 03:23:19.902652 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: Bootstrap starting.
I20260927 03:23:19.903532 7210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:19.903784 7210 log.cc:824] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:19.904441 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: No bootstrap required, opened a new log
I20260927 03:23:19.905885 7210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } }
I20260927 03:23:19.906036 7210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:19.906068 7210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c20d31f9fb64676a11df189db754b54, State: Initialized, Role: FOLLOWER
I20260927 03:23:19.906196 7210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } }
I20260927 03:23:19.906477 7210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:19.906454 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } } }
I20260927 03:23:19.906523 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:23:19.907960 7226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:23:19.908015 7226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:23:19.910156 7142 master_runner.cc:428] Detected that this master 2c20d31f9fb64676a11df189db754b54 is joining an existing cluster
I20260927 03:23:19.910204 7142 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:33087
I20260927 03:23:19.919090 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.911728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:43806)
0927 03:23:19.911921 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.911930 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.912036 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.912210 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.912211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.912533 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.912600 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.912602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.913567 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.913568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.914464 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.914467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.914983 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.914984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.915056 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.915562 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.915566 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.916010 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.917866 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.917867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.917868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.918095 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.918351 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.918352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.918352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.918435 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.918740 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.918740 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.918783 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.918883 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.918985 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260927 03:23:19.919106 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.911676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43806 (local address 127.0.98.62:37939)
0927 03:23:19.911725 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.911727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.911989 (+ 262us) server_negotiation.cc:407] Connection header received
0927 03:23:19.912111 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.912112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.912124 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.912152 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.912693 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.913439 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.914557 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.914866 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.914909 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.916086 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.916087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.916088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.916095 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.917739 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.918180 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.918181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.918181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.918249 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.918521 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.918522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.918522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.918656 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.918698 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.918939 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.919019 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479398, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 03:23:19.936182 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.911676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.0.98.61:33793)
0927 03:23:19.911760 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.911762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.912035 (+ 273us) server_negotiation.cc:407] Connection header received
0927 03:23:19.912117 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.912118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.912128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.912155 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.912671 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.913482 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.914772 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.915085 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.915127 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.918740 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.918742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.918742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.918750 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.920205 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.935271 (+ 15066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.935273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.935273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.935342 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.935576 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.935577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.935578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.935724 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.935768 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.935992 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.936082 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 03:23:19.936501 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.911808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33793 (local address 127.0.0.1:55920)
0927 03:23:19.911982 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.911990 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.912055 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.912209 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.912209 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.912512 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.912583 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.912585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.913592 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.913593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.914671 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.914673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.917823 (+ 3150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.917824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.917912 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.918350 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.918354 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.918658 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.920414 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.920415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.920415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.935174 (+ 14759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.935405 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.935405 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.935406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.935473 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.935817 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.935818 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.935866 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.936279 (+ 413us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.936399 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":1,"thread_start_us":51,"threads_started":1}
I20260927 03:23:19.938473 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.921288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33087 (local address 127.0.0.1:45754)
0927 03:23:19.921343 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.921353 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.921386 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.921626 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.921627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.921741 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.921777 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.921778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.922722 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.922723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.923292 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.923294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.923739 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.923740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.923790 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.924138 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.924141 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.925352 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.937391 (+ 12039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.937393 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.937394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.937621 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.937829 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.937830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.937830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.937895 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.938149 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.938150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.938204 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.938264 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.938371 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 03:23:19.938481 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.921367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45754 (local address 127.0.98.60:33087)
0927 03:23:19.921502 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.921504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.921510 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:23:19.921542 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.921542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.921552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.921611 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.921840 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.922630 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.923357 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.923676 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.923740 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.925427 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.925428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.925428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.925435 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.937286 (+ 11851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.937696 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.937698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.937698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.937764 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.937956 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.937957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.937957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.938087 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.938145 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.938297 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.938369 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260927 03:23:19.938861 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.911833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45744 (local address 127.0.98.60:33087)
0927 03:23:19.911866 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.911868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.912005 (+ 137us) server_negotiation.cc:407] Connection header received
0927 03:23:19.912114 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.912115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.912123 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.912148 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.912802 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.913512 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.914498 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.914771 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.914806 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.935964 (+ 21158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.935965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.935966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.935972 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.937802 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.938098 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.938098 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.938099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.938147 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.938324 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.938325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.938325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.938424 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.938456 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.938623 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.938740 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":16}
I20260927 03:23:19.938871 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.911670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33087 (local address 127.0.0.1:45744)
0927 03:23:19.911920 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.911937 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.912051 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.912200 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.912201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.912685 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.912735 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.912737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.913592 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.913593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.914421 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.914424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.914973 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.914974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.915034 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.935477 (+ 20443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.935482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.935887 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.937873 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.937873 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.937873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.938044 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.938200 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.938200 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.938201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.938253 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.938491 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.938491 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.938523 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.938576 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.938686 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260927 03:23:19.939476 6489 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:33087: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2c20d31f9fb64676a11df189db754b54"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 33087
}
attrs {
promote: true
}
}
dest_uuid: "4216573a82c74be9b92dbe0e0046c1ba"
cas_config_opid_index: 5
I20260927 03:23:19.939702 6489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: true } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } }
I20260927 03:23:19.940161 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2c20d31f9fb64676a11df189db754b54 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: true } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } } }
I20260927 03:23:19.940258 7184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:19.941159 7048 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:33087
I20260927 03:23:19.941232 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: true } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } } }
I20260927 03:23:19.941296 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:19.941490 7142 master.cc:537] Master@127.0.98.60:33087 shutting down...
I20260927 03:23:19.941639 7184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 1 FOLLOWER]: Refusing update from remote peer 4216573a82c74be9b92dbe0e0046c1ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 03:23:19.941963 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:23:19.942603 6473 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba -> Peer 2c20d31f9fb64676a11df189db754b54 (127.0.98.60:33087): Couldn't send request to peer 2c20d31f9fb64676a11df189db754b54. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 03:23:19.944799 7142 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:19.944896 7142 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:19.944937 7142 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: stopping tablet replica
I20260927 03:23:19.945708 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.940182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.0.98.61:33793)
0927 03:23:19.940217 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.940220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.941542 (+ 1322us) server_negotiation.cc:407] Connection header received
0927 03:23:19.941566 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.941567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.941591 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.941645 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:19.941952 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.942624 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.944049 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.944614 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.944685 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.944772 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.945600 (+ 828us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.945645 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:23:19.945073 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.940121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33793 (local address 127.0.0.1:55936)
0927 03:23:19.941441 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.941452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.941494 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.941843 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.941845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.941847 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:19.941891 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.941893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.942701 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.942702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.943972 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.943975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.944806 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.944807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.944873 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.944897 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.944956 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.945000 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1295}
I20260927 03:23:19.946028 7098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 LEARNER]: Refusing update from remote peer 4216573a82c74be9b92dbe0e0046c1ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260927 03:23:19.946204 7160 connection.cc:441] server connection from 127.0.0.1:45726 torn down before Call kudu.master.MasterService.ListTables from 127.0.0.1:45726 (request call id 22) could send its response
I20260927 03:23:19.946223 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 03:23:19.946806 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2c20d31f9fb64676a11df189db754b54 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: true } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } } }
I20260927 03:23:19.947715 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4216573a82c74be9b92dbe0e0046c1ba. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: true } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } } }
I20260927 03:23:19.947815 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:23:19.947734 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: true } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } } }
I20260927 03:23:19.947968 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:23:19.948211 7049 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba: attempting to promote NON_VOTER af86687a976b42eeb8255db2ef798344 to VOTER
I20260927 03:23:19.948398 7049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } }
I20260927 03:23:19.948761 7098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 LEARNER]: Refusing update from remote peer 4216573a82c74be9b92dbe0e0046c1ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 03:23:19.948889 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.948707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33087 (local address 127.0.0.1:45756)
0927 03:23:19.948759 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.948857 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 03:23:19.948933 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 03:23:19.949280 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.949202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33087 (local address 127.0.0.1:45772)
0927 03:23:19.949231 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.949258 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260927 03:23:19.949347 6473 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba -> Peer 2c20d31f9fb64676a11df189db754b54 (127.0.98.60:33087): Couldn't send request to peer 2c20d31f9fb64676a11df189db754b54. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 03:23:19.949529 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, af86687a976b42eeb8255db2ef798344 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } } }
I20260927 03:23:19.949697 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, af86687a976b42eeb8255db2ef798344 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } } }
I20260927 03:23:19.950392 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } } }
I20260927 03:23:19.950399 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } } }
I20260927 03:23:19.950469 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:23:19.950471 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:19.950516 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } } }
I20260927 03:23:19.950563 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:19.958981 7142 master.cc:552] Master@127.0.98.60:33087 shutdown complete.
I20260927 03:23:19.959038 7142 master_runner.cc:305] Clearing existing system tablet
I20260927 03:23:19.959158 7142 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:19.959203 7142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:19.959770 7142 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:19.960235 7239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.960336 7142 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:19.960386 7142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.960407 7142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/wal
uuid: "2c20d31f9fb64676a11df189db754b54"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "cae6d33c8efef16d0421bf01ba331ac4"
server_key_iv: "1d5078462d3932d4e901fa91258bfba9"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.961030 7142 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:19.962500 7142 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 03:23:19.962572 7142 log.cc:1196] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 03:23:19.962744 7142 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: Deleting consensus metadata
I20260927 03:23:19.962898 7142 master_runner.cc:315] Copying system tablet from 127.0.98.62:37939
I20260927 03:23:19.963454 7142 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37939
I20260927 03:23:19.969148 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.963821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.0.98.62:37939)
0927 03:23:19.963893 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:19.963895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:19.964000 (+ 105us) server_negotiation.cc:407] Connection header received
0927 03:23:19.964070 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:19.964071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:19.964081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:19.964128 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:19.964438 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.964927 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.965672 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.965878 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.965932 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.966849 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:19.966851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:19.966851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:19.966860 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:19.968151 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.968460 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.968461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.968462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.968517 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:19.968668 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:19.968669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:19.968669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:19.968780 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:19.968836 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:19.968976 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 03:23:19.969052 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:23:19.969151 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:19.963744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:43814)
0927 03:23:19.963925 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:19.963936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:19.964009 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:19.964131 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:19.964132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:19.964296 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:19.964362 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.964364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.964996 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.964997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.965589 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:19.965591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:19.965938 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:19.965939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:19.966002 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:19.966445 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:19.966450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:19.966740 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:19.968234 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.968235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.968235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.968401 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.968566 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:19.968567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:19.968567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:19.968621 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:19.968835 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:19.968836 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:19.968884 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 03:23:19.968944 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:23:19.969052 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260927 03:23:19.969420 6508 tablet_copy_service.cc:140] P 4216573a82c74be9b92dbe0e0046c1ba: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2c20d31f9fb64676a11df189db754b54 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43814)
I20260927 03:23:19.969477 6508 tablet_copy_service.cc:161] P 4216573a82c74be9b92dbe0e0046c1ba: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2c20d31f9fb64676a11df189db754b54 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43814: session id = 2c20d31f9fb64676a11df189db754b54-00000000000000000000000000000000
I20260927 03:23:19.969918 6508 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 03:23:19.970445 7142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 03:23:19.971504 7142 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: tablet copy: Starting download of 0 data blocks...
I20260927 03:23:19.971627 7142 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: tablet copy: Starting download of 1 WAL segments...
I20260927 03:23:19.972326 7142 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 03:23:19.973129 6508 tablet_copy_service.cc:342] P 4216573a82c74be9b92dbe0e0046c1ba: Request end of tablet copy session 2c20d31f9fb64676a11df189db754b54-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43814
I20260927 03:23:19.973181 6508 tablet_copy_service.cc:433] P 4216573a82c74be9b92dbe0e0046c1ba: ending tablet copy session 2c20d31f9fb64676a11df189db754b54-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2c20d31f9fb64676a11df189db754b54
I20260927 03:23:19.975412 7142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:19.976701 7246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:19.976706 7142 server_base.cc:1034] running on GCE node
W20260927 03:23:19.976716 7247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:19.976953 7249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:19.977093 7142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:19.977208 7142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:19.978327 7142 hybrid_clock.cc:648] HybridClock initialized: now 1790479399978318 us; error 24 us; skew 500 ppm
Sep 27 03:23:19 dist-test-slave-gw2v krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479399, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:19.979555 7142 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 03:23:19.979792 7142 webserver.cc:460] Webserver started at http://127.0.98.60:46253/ using document root <none> and password file <none>
I20260927 03:23:19.979897 7142 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:19.979929 7142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:19.980458 7142 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.980912 7256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.981034 7142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:19.981076 7142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/wal
uuid: "2c20d31f9fb64676a11df189db754b54"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "cae6d33c8efef16d0421bf01ba331ac4"
server_key_iv: "1d5078462d3932d4e901fa91258bfba9"
server_key_version: "encryptionkey@0"
I20260927 03:23:19.981129 7142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:20.001006 7142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:20.002535 7142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:20.005250 7142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33087
I20260927 03:23:20.005450 7308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33087 every 8 connection(s)
I20260927 03:23:20.005578 7142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/master-2/data/info.pb
I20260927 03:23:20.006453 7309 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:23:20.007035 7309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: Bootstrap starting.
I20260927 03:23:20.010481 7309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:23:20.010715 7309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: Bootstrap complete.
I20260927 03:23:20.010962 7309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } }
I20260927 03:23:20.011059 7309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c20d31f9fb64676a11df189db754b54, State: Initialized, Role: LEARNER
I20260927 03:23:20.011118 7309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } }
I20260927 03:23:20.011314 7309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:20.011292 7312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } } }
I20260927 03:23:20.011363 7312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:23:20.012712 7323 catalog_manager.cc:1259] Loaded cluster ID: 5476a318fb324d70a21b536467118dfc
I20260927 03:23:20.012794 7323 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: loading cluster ID for follower catalog manager: success
I20260927 03:23:20.013888 7323 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: acquiring CA information for follower catalog manager: success
I20260927 03:23:20.014168 7323 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:23:20.054106 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.047657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.0.98.60:33087)
0927 03:23:20.047764 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.047766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.047776 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:20.047799 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.047800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.047808 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.047871 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:20.048539 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.049184 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.050128 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.050321 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.050370 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.051662 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:20.051663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:20.051664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:20.051672 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:20.053027 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.053374 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.053375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.053375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.053441 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.053581 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.053581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.053582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.053700 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:20.053787 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.053921 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.054006 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
I20260927 03:23:20.054637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 6551
I20260927 03:23:20.059034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42473
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37939
--webserver_port=35921
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37939,127.0.98.61:33793
--tserver_master_addrs=127.0.98.62:37939,127.0.98.61:33793,127.0.98.60:33087 with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:20.140179 7325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:20.140396 7325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:20.140419 7325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:20.141577 7325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:20.141623 7325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:20.141649 7325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:23:20.143153 7325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35921
--tserver_master_addrs=127.0.98.62:37939,127.0.98.61:33793,127.0.98.60:33087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:20.143406 7325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:20.143649 7325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:20.148756 7333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:20.148753 7331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:20.148885 7325 server_base.cc:1034] running on GCE node
W20260927 03:23:20.148778 7330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:20.149286 7325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:20.149541 7325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:20.150682 7325 hybrid_clock.cc:648] HybridClock initialized: now 1790479400150675 us; error 29 us; skew 500 ppm
Sep 27 03:23:20 dist-test-slave-gw2v krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479400, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:20.153235 7325 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 03:23:20.153683 7325 webserver.cc:460] Webserver started at http://127.0.98.1:35921/ using document root <none> and password file <none>
I20260927 03:23:20.153887 7325 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:20.153944 7325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:20.155321 7325 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:20.155849 7340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:20.155995 7325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:23:20.156077 7325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "387121f928344d86a04295c84ae74dd7"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "570830e5ba11302dd502edf173a00c29"
server_key_iv: "4b1d1ffd8f574190c02ce561aa81b807"
server_key_version: "encryptionkey@0"
I20260927 03:23:20.156400 7325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:20.172169 7325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:20.174165 7325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:20.174280 7325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:20.174475 7325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:20.174772 7325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:20.174806 7325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:20.174857 7325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:20.174883 7325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:20.180388 7325 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42473
I20260927 03:23:20.180444 7453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42473 every 8 connection(s)
I20260927 03:23:20.180775 7325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:23:20.183823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 7325
I20260927 03:23:20.183970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 6685
Sep 27 03:23:20 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479400, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 03:23:20.189924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36333
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37939
--webserver_port=36647
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37939,127.0.98.61:33793
--tserver_master_addrs=127.0.98.62:37939,127.0.98.61:33793,127.0.98.60:33087 with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:23:20 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479400, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 03:23:20 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479400, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:23:20.193617 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.181754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40525 (local address 127.0.98.61:33793)
0927 03:23:20.181798 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.181801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.181812 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:23:20.182794 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.182795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.182806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.182855 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:20.183605 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.184357 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.188550 (+ 4193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.188806 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.188873 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.190854 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:20.190856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:20.190856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:20.190863 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:20.192359 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.192728 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.192729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.192729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.192807 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.193002 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.193003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.193003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.193132 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:20.193187 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.193454 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.193528 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:23:20.193818 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.181504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33793 (local address 127.0.98.1:40525)
0927 03:23:20.181694 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:20.181738 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:20.182671 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:20.182926 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:20.182931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:20.183288 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:20.183491 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.183498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.187666 (+ 4168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.187666 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.188440 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.188443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.188888 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.188889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.188961 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.189370 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:20.189375 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:20.190738 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:20.192434 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.192435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.192435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.192656 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.192872 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.192873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.192873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.192929 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.193190 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:20.193197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:20.193337 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 03:23:20.193426 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 03:23:20.193593 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":30,"thread_start_us":96,"threads_started":1}
I20260927 03:23:20.194307 7456 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33793
I20260927 03:23:20.194415 7456 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:20.194602 7456 heartbeater.cc:511] Master 127.0.98.61:33793 requested a full tablet report, sending...
I20260927 03:23:20.195120 7088 ts_manager.cc:194] Registered new tserver with Master: 387121f928344d86a04295c84ae74dd7 (127.0.98.1:42473)
I20260927 03:23:20.195257 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.181811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53471 (local address 127.0.98.60:33087)
0927 03:23:20.181898 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.181900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.181927 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:23:20.182795 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.182796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.182810 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.182865 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:20.183622 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.184319 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.185543 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.185800 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.185848 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.189683 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:20.189684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:20.189685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:20.189692 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:20.192139 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.192651 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.192652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.192652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.192707 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.192967 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.192968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.192968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.194479 (+ 1511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:20.194523 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.195072 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.195166 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 03:23:20.195592 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.181504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33087 (local address 127.0.98.1:53471)
0927 03:23:20.181869 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:20.181877 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:20.182686 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:20.182926 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:20.182931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:20.183284 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:20.183491 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.183498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.184591 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.184592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.185432 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.185434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.185906 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.185907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.186072 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.187825 (+ 1753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:20.187835 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:20.189580 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:20.192230 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.192237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.192240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.192579 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.192769 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.192769 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.192770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.192893 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.194573 (+ 1680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:20.194574 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:20.194617 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:23:20.195373 (+ 756us) client_negotiation.cc:240] Negotiation successful
0927 03:23:20.195475 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":28,"thread_start_us":90,"threads_started":1}
I20260927 03:23:20.196090 7454 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33087
I20260927 03:23:20.196139 7454 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:20.196200 7454 heartbeater.cc:511] Master 127.0.98.60:33087 requested a full tablet report, sending...
I20260927 03:23:20.196568 7273 ts_manager.cc:194] Registered new tserver with Master: 387121f928344d86a04295c84ae74dd7 (127.0.98.1:42473)
I20260927 03:23:20.197058 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.181753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33933 (local address 127.0.98.62:37939)
0927 03:23:20.181860 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.181862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.182482 (+ 620us) server_negotiation.cc:407] Connection header received
0927 03:23:20.182755 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.182756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.182768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.182834 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:20.183580 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.184356 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.190413 (+ 6057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.190715 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.190760 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.192727 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:20.192728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:20.192729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:20.192737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:20.194264 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.195720 (+ 1456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.195721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.195721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.195781 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.196167 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.196168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.196169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.196305 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:20.196368 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.196916 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.197000 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 03:23:20.197103 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.181504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.1:33933)
0927 03:23:20.182000 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:20.182009 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:20.182676 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:20.182926 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:20.182929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:20.183277 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:20.183491 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.183496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.189549 (+ 6053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.189550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.190303 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.190306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.190858 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.190859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.190925 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.191228 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:20.191233 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:20.192628 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:20.194691 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.194692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.194692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.194965 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.195954 (+ 989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.195954 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.195955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.196088 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.196762 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:20.196763 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:20.196818 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 03:23:20.196880 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 03:23:20.196988 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":27,"thread_start_us":76,"threads_started":1}
I20260927 03:23:20.197446 7455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37939
I20260927 03:23:20.197504 7455 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:20.197557 7455 heartbeater.cc:511] Master 127.0.98.62:37939 requested a full tablet report, sending...
I20260927 03:23:20.197795 6489 ts_manager.cc:194] Re-registered known tserver with Master: 387121f928344d86a04295c84ae74dd7 (127.0.98.1:42473)
I20260927 03:23:20.198156 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33933
W20260927 03:23:20.272869 7461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:20.273113 7461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:20.273145 7461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:20.274317 7461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:20.274364 7461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:20.274407 7461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:23:20.275969 7461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36647
--tserver_master_addrs=127.0.98.62:37939,127.0.98.61:33793,127.0.98.60:33087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:20.276214 7461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:20.276428 7461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:20.281559 7467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:20.281559 7469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:20.281577 7461 server_base.cc:1034] running on GCE node
W20260927 03:23:20.281603 7466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:20.282014 7461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:20.282337 7461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:20.283558 7461 hybrid_clock.cc:648] HybridClock initialized: now 1790479400283539 us; error 32 us; skew 500 ppm
Sep 27 03:23:20 dist-test-slave-gw2v krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479400, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:20.286331 7461 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 03:23:20.286886 7461 webserver.cc:460] Webserver started at http://127.0.98.2:36647/ using document root <none> and password file <none>
I20260927 03:23:20.287114 7461 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:20.287192 7461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:20.288574 7461 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:20.289232 7476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:20.289440 7461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:20.289535 7461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "ef4d9fd028fb4283b48cc865333cb3ca"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "c1440a728ac92779a650d0bf819be425"
server_key_iv: "70af8e60acbf7cc750557d96fc40cb84"
server_key_version: "encryptionkey@0"
I20260927 03:23:20.289892 7461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:20.311655 7461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:20.314288 7461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:20.314474 7461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:20.314757 7461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:20.315092 7461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:20.315127 7461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:20.315155 7461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:20.315172 7461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:20.320703 7461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36333
I20260927 03:23:20.320757 7589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36333 every 8 connection(s)
I20260927 03:23:20.321151 7461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:23:20.325591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 7461
I20260927 03:23:20.325742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 6819
Sep 27 03:23:20 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479400, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 03:23:20 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479400, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 03:23:20 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479400, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 03:23:20.331717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45035
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37939
--webserver_port=46669
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37939,127.0.98.61:33793
--tserver_master_addrs=127.0.98.62:37939,127.0.98.61:33793,127.0.98.60:33087 with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:23:20.332933 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.322135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47443 (local address 127.0.98.60:33087)
0927 03:23:20.322178 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.322180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.322320 (+ 140us) server_negotiation.cc:407] Connection header received
0927 03:23:20.322802 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.322802 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.322812 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.322962 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:20.323980 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.324491 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.325765 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.325996 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.326029 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.329736 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:20.329737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:20.329738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:20.329745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:20.331403 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.331907 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.331908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.331908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.331965 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.332237 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.332238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.332238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.332364 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:20.332402 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.332749 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.332829 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:23:20.332937 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.321921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54901 (local address 127.0.98.62:37939)
0927 03:23:20.322037 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.322038 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.322954 (+ 916us) server_negotiation.cc:407] Connection header received
0927 03:23:20.322997 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.322998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.323010 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.323061 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:20.323909 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.324735 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.325682 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.325964 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.326006 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.329315 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:20.329317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:20.329318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:20.329326 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:20.331310 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.331907 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.331909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.331910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.331971 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.332255 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.332256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.332256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.332381 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:20.332422 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.332763 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.332847 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 03:23:20.333523 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.321858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33087 (local address 127.0.98.2:47443)
0927 03:23:20.322209 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:20.322245 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:20.322712 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:20.322973 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:20.322977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:20.323616 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:20.323828 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.323835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.324584 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.324585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.325670 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.325673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.326101 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.326102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.326228 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.326984 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:20.326993 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:20.329641 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:20.331490 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.331491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.331491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.331813 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.332029 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.332030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.332030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.332173 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.332443 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:20.332448 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:20.332610 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 03:23:20.333024 (+ 414us) client_negotiation.cc:240] Negotiation successful
0927 03:23:20.333249 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":45,"thread_start_us":124,"threads_started":1}
I20260927 03:23:20.333529 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.321858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.2:54901)
0927 03:23:20.322887 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:20.322896 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:20.322948 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:20.323071 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:20.323073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:20.323558 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:20.323829 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.323834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.324831 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.324832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.325580 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.325583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.326536 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.326537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.326601 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.327325 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:20.327330 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:20.329215 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:20.331397 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.331405 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.331408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.331811 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.332043 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.332043 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.332044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.332176 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.332477 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:20.332477 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:20.332609 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 03:23:20.333024 (+ 415us) client_negotiation.cc:240] Negotiation successful
0927 03:23:20.333323 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":112,"thread_start_us":115,"threads_started":1}
I20260927 03:23:20.334036 7590 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33087
I20260927 03:23:20.334139 7590 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:20.334141 7591 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37939
I20260927 03:23:20.334192 7591 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:20.334252 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.322081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44893 (local address 127.0.98.61:33793)
0927 03:23:20.322146 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.322148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.323085 (+ 937us) server_negotiation.cc:407] Connection header received
0927 03:23:20.323170 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.323172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.323182 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.323230 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:20.323945 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.324705 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.326862 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.327174 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.327243 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.329736 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:20.329737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:20.329738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:20.329746 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:20.332098 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.332637 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.332638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.332639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.332695 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.332876 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.332877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.332877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.333008 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:20.333050 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.334112 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.334193 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 03:23:20.334338 7590 heartbeater.cc:511] Master 127.0.98.60:33087 requested a full tablet report, sending...
I20260927 03:23:20.334338 7591 heartbeater.cc:511] Master 127.0.98.62:37939 requested a full tablet report, sending...
I20260927 03:23:20.334762 6489 ts_manager.cc:194] Re-registered known tserver with Master: ef4d9fd028fb4283b48cc865333cb3ca (127.0.98.2:36333)
I20260927 03:23:20.334851 7273 ts_manager.cc:194] Registered new tserver with Master: ef4d9fd028fb4283b48cc865333cb3ca (127.0.98.2:36333)
I20260927 03:23:20.335177 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54901
I20260927 03:23:20.339174 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.321858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33793 (local address 127.0.98.2:44893)
0927 03:23:20.323034 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:20.323042 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:20.323106 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:20.323242 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:20.323243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:20.323592 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:20.323828 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.323835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.324779 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.324780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.326756 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.326758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.327256 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.327257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.327325 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.327680 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:20.327685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:20.329641 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:20.332375 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.332376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.332376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.332561 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.332761 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.332761 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.332761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.332813 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.333465 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:20.333466 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:20.333509 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:23:20.334527 (+ 1018us) client_negotiation.cc:240] Negotiation successful
0927 03:23:20.334631 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1142,"mutex_wait_us":65,"thread_start_us":107,"threads_started":1}
I20260927 03:23:20.339568 7592 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33793
I20260927 03:23:20.339633 7592 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:20.339720 7592 heartbeater.cc:511] Master 127.0.98.61:33793 requested a full tablet report, sending...
I20260927 03:23:20.340015 7088 ts_manager.cc:194] Registered new tserver with Master: ef4d9fd028fb4283b48cc865333cb3ca (127.0.98.2:36333)
W20260927 03:23:20.414204 7597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:20.414507 7597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:20.414549 7597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:20.415946 7597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:20.416108 7597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:20.416146 7597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:23:20.417901 7597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46669
--tserver_master_addrs=127.0.98.62:37939,127.0.98.61:33793,127.0.98.60:33087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:20.418123 7597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:20.418322 7597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:20.423524 7605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:20.423556 7602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:20.423693 7597 server_base.cc:1034] running on GCE node
W20260927 03:23:20.423524 7603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:20.424044 7597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:20.424384 7597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:20.425557 7597 hybrid_clock.cc:648] HybridClock initialized: now 1790479400425543 us; error 31 us; skew 500 ppm
Sep 27 03:23:20 dist-test-slave-gw2v krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479400, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:20.428296 7597 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 03:23:20.428778 7597 webserver.cc:460] Webserver started at http://127.0.98.3:46669/ using document root <none> and password file <none>
I20260927 03:23:20.429005 7597 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:20.429069 7597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:20.430495 7597 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:20.431363 7612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:20.431577 7597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:23:20.431663 7597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "bbd404a350704e0483c7b2f08f07d77e"
format_stamp: "Formatted at 2026-09-27 03:23:19 on dist-test-slave-gw2v"
server_key: "fc7a4dbd24d188808990e3251799f815"
server_key_iv: "7d5a4cf550e218cc2f577ee856d0bfca"
server_key_version: "encryptionkey@0"
I20260927 03:23:20.432054 7597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:20.465554 7597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:20.467634 7597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:20.467758 7597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:20.468005 7597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:20.468302 7597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:20.468346 7597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:20.468384 7597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:20.468406 7597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:20.474047 7597 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45035
I20260927 03:23:20.474098 7725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45035 every 8 connection(s)
I20260927 03:23:20.474442 7597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:23:20.478638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 7597
Sep 27 03:23:20 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479400, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 03:23:20 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479400, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 03:23:20 dist-test-slave-gw2v krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479400, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:23:20.487260 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.475249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37037 (local address 127.0.98.60:33087)
0927 03:23:20.475429 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.475430 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.475443 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:23:20.476576 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.476578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.476590 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.476620 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:20.477512 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.478075 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.479567 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.479844 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.479881 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.483442 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:20.483443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:20.483444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:20.483452 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:20.485264 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.486072 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.486074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.486074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.486141 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.486416 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.486417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.486417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.486546 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:20.486580 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.487092 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.487171 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260927 03:23:20.487326 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.475239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60569 (local address 127.0.98.62:37939)
0927 03:23:20.475290 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.475292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.475738 (+ 446us) server_negotiation.cc:407] Connection header received
0927 03:23:20.476577 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.476578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.476590 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.476620 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:20.477505 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.478047 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.479585 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.479846 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.479901 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.483609 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:20.483612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:20.483612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:20.483619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:20.485513 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.486072 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.486074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.486074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.486141 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.486416 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.486417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.486417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.486667 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:20.486722 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.487092 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.487233 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:23:20.487643 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.475169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.3:60569)
0927 03:23:20.475678 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:20.475687 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:20.476462 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:20.476705 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:20.476705 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:20.477143 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:20.477404 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.477411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.478146 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.478147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.479473 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.479476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.479981 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.479982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.480121 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.481193 (+ 1072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:20.481198 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:20.483516 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:20.485602 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.485609 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.485611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.485989 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.486207 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.486208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.486208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.486341 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.486731 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:20.486738 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:20.486935 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 03:23:20.487215 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 03:23:20.487396 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":36,"spinlock_wait_cycles":8064,"thread_start_us":93,"threads_started":1}
I20260927 03:23:20.488123 7727 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37939
I20260927 03:23:20.488234 7727 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:20.488399 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.475168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33087 (local address 127.0.98.3:37037)
0927 03:23:20.475336 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:20.475378 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:20.476455 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:20.476694 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:20.476700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:20.477148 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:20.477404 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.477411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.478162 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.478163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.479472 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.479474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.479973 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.479974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.480121 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.480841 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:20.480851 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:20.483240 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:20.485643 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.485644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.485644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.485989 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.486203 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.486204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.486204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.486341 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.486764 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:20.486764 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:20.486935 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 03:23:20.488112 (+ 1177us) client_negotiation.cc:240] Negotiation successful
0927 03:23:20.488258 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":68,"thread_start_us":87,"threads_started":1}
I20260927 03:23:20.488458 7727 heartbeater.cc:511] Master 127.0.98.62:37939 requested a full tablet report, sending...
I20260927 03:23:20.489168 6489 ts_manager.cc:194] Re-registered known tserver with Master: bbd404a350704e0483c7b2f08f07d77e (127.0.98.3:45035)
I20260927 03:23:20.489451 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.475556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42793 (local address 127.0.98.61:33793)
0927 03:23:20.475598 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.475600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.476229 (+ 629us) server_negotiation.cc:407] Connection header received
0927 03:23:20.476545 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.476546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.476558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.476605 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:20.478130 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.478865 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.479809 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.480070 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.480108 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.483059 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:20.483061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:20.483061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:20.483068 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:20.486621 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.488678 (+ 2057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.488680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.488680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.488735 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.488936 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.488937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.488937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.489058 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:20.489097 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.489288 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.489355 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:23:20.489596 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60569
I20260927 03:23:20.489744 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.475456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33793 (local address 127.0.98.3:42793)
0927 03:23:20.476162 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:20.476171 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:20.476459 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:20.476987 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:20.476988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:20.477173 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:20.477402 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.477407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.478955 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.478956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.479728 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.479729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.480177 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.480178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.480230 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.481084 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:20.481088 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:20.482886 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:20.488022 (+ 5136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.488023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.488023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.488595 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.488804 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:20.488805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:20.488805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:20.488864 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:20.489143 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:20.489144 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:20.489185 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 03:23:20.489551 (+ 366us) client_negotiation.cc:240] Negotiation successful
0927 03:23:20.489648 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":22,"thread_start_us":215,"threads_started":1}
I20260927 03:23:20.490056 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33793
I20260927 03:23:20.490093 7728 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:20.490433 7728 heartbeater.cc:511] Master 127.0.98.61:33793 requested a full tablet report, sending...
I20260927 03:23:20.490733 7088 ts_manager.cc:194] Registered new tserver with Master: bbd404a350704e0483c7b2f08f07d77e (127.0.98.3:45035)
I20260927 03:23:20.490922 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.480237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43820 (local address 127.0.98.62:37939)
0927 03:23:20.480698 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.480700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.480707 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:23:20.480725 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.480725 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.480736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.480786 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:20.481640 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.482229 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.483275 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.483548 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.483599 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.485025 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:20.485026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:20.485026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:20.485033 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:20.487948 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.488846 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.488847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.488847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.488913 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.489515 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.489516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.489516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.490321 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:20.490409 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.490728 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.490808 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":50,"threads_started":1}
I20260927 03:23:20.491571 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.479846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55944 (local address 127.0.98.61:33793)
0927 03:23:20.480521 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.480523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.480562 (+ 39us) server_negotiation.cc:407] Connection header received
0927 03:23:20.480622 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.480622 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.480634 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.480707 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:20.482576 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.483313 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.484157 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.484451 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.484511 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.485798 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:20.485800 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:20.485800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:20.485807 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:20.490197 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.490557 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.490558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.490558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.490609 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.490898 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.490899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.490899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.491016 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:20.491101 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.491415 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.491486 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":53,"threads_started":1}
I20260927 03:23:20.492058 7726 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33087
I20260927 03:23:20.492107 7726 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:20.492162 7726 heartbeater.cc:511] Master 127.0.98.60:33087 requested a full tablet report, sending...
I20260927 03:23:20.492389 7273 ts_manager.cc:194] Registered new tserver with Master: bbd404a350704e0483c7b2f08f07d77e (127.0.98.3:45035)
I20260927 03:23:20.493618 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.484911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.0.98.60:33087)
0927 03:23:20.486183 (+ 1272us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.486185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.486330 (+ 145us) server_negotiation.cc:407] Connection header received
0927 03:23:20.486350 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.486351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.486362 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.486433 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:20.486975 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.487770 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.488761 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.489031 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.489086 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.490846 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:20.490847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:20.490848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:20.490854 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:20.492533 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.492916 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.492917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.492918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.492983 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.493142 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.493143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.493143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.493279 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:20.493370 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.493459 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.493536 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1251,"thread_start_us":55,"threads_started":1}
I20260927 03:23:20.493166 6489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:23:20.495141 6489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:23:20.502465 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.498749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.0.98.2:36333)
0927 03:23:20.498919 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.498924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.499435 (+ 511us) server_negotiation.cc:407] Connection header received
0927 03:23:20.499474 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.499476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.499490 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.499556 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:20.499604 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.500608 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.501639 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.502130 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.502202 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.502277 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.502355 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.502394 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260927 03:23:20.502453 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.498661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36333 (local address 127.0.0.1:33156)
0927 03:23:20.499331 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:20.499339 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:20.499386 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:20.499548 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:20.499549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:20.499550 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:20.499569 (+ 19us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.499570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.500680 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.500680 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.501563 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.501565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.502214 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.502215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.502284 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.502288 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:20.502340 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:23:20.502379 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":85,"threads_started":1}
I20260927 03:23:20.503105 7524 tablet_service.cc:1467] Processing CreateTablet for tablet 164f1fcd75234d0496a466ac73021b62 (DEFAULT_TABLE table=test-table [id=0cc3f02e436447cd979cb31d62183687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:20.503504 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164f1fcd75234d0496a466ac73021b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:20.503652 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.498801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.0.98.1:42473)
0927 03:23:20.498974 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.498979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.499142 (+ 163us) server_negotiation.cc:407] Connection header received
0927 03:23:20.499186 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.499188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.499204 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.499269 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:20.499389 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.501023 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.502310 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.503332 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.503394 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.503463 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.503538 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.503577 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260927 03:23:20.503837 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.498613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42473 (local address 127.0.0.1:59198)
0927 03:23:20.499025 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:20.499036 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:20.499089 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:20.499284 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:20.499285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:20.499287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:20.499325 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.499326 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.501100 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.501101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.502218 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.502220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.503394 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.503395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.503454 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.503459 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:23:20.503518 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 03:23:20.503629 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":70,"threads_started":1}
I20260927 03:23:20.504477 7388 tablet_service.cc:1467] Processing CreateTablet for tablet 164f1fcd75234d0496a466ac73021b62 (DEFAULT_TABLE table=test-table [id=0cc3f02e436447cd979cb31d62183687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:20.504813 7388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164f1fcd75234d0496a466ac73021b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:20.504827 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.498676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45035 (local address 127.0.0.1:34048)
0927 03:23:20.500734 (+ 2058us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:20.500745 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:20.500799 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:20.501063 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:20.501064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:20.501066 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:20.501101 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.501102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.502310 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.502311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.503729 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.503732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.504577 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.504578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.504645 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.504650 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:23:20.504716 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:23:20.504752 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2037,"thread_start_us":75,"threads_started":1}
I20260927 03:23:20.504841 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.498904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34048 (local address 127.0.98.3:45035)
0927 03:23:20.499037 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.499041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.500810 (+ 1769us) server_negotiation.cc:407] Connection header received
0927 03:23:20.500913 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.500915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.500931 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.501004 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:20.501157 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.502221 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.503793 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.504483 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.504556 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.504622 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.504730 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.504773 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260927 03:23:20.507536 7756 tablet_bootstrap.cc:492] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7: Bootstrap starting.
I20260927 03:23:20.507799 7660 tablet_service.cc:1467] Processing CreateTablet for tablet 164f1fcd75234d0496a466ac73021b62 (DEFAULT_TABLE table=test-table [id=0cc3f02e436447cd979cb31d62183687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:20.508036 7660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164f1fcd75234d0496a466ac73021b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:20.508086 7756 tablet_bootstrap.cc:654] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:20.508706 7756 log.cc:824] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:20.509438 7756 tablet_bootstrap.cc:492] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7: No bootstrap required, opened a new log
I20260927 03:23:20.509503 7756 ts_tablet_manager.cc:1397] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:23:20.510486 7758 tablet_bootstrap.cc:492] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e: Bootstrap starting.
I20260927 03:23:20.511092 7758 tablet_bootstrap.cc:654] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:20.511032 7756 raft_consensus.cc:348] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } }
I20260927 03:23:20.511155 7756 raft_consensus.cc:374] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:20.511186 7756 raft_consensus.cc:729] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387121f928344d86a04295c84ae74dd7, State: Initialized, Role: FOLLOWER
I20260927 03:23:20.511188 7755 tablet_bootstrap.cc:492] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca: Bootstrap starting.
I20260927 03:23:20.511313 7756 consensus_queue.cc:260] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } }
I20260927 03:23:20.511430 7758 log.cc:824] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:20.511560 7756 ts_tablet_manager.cc:1428] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:23:20.511650 7455 heartbeater.cc:503] Master 127.0.98.62:37939 was elected leader, sending a full tablet report...
I20260927 03:23:20.512116 7758 tablet_bootstrap.cc:492] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e: No bootstrap required, opened a new log
I20260927 03:23:20.512180 7758 ts_tablet_manager.cc:1397] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:23:20.512622 7755 tablet_bootstrap.cc:654] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:20.513209 7755 log.cc:824] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:20.513916 7755 tablet_bootstrap.cc:492] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca: No bootstrap required, opened a new log
I20260927 03:23:20.513985 7755 ts_tablet_manager.cc:1397] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:23:20.514035 7758 raft_consensus.cc:348] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } }
I20260927 03:23:20.514185 7758 raft_consensus.cc:374] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:20.514222 7758 raft_consensus.cc:729] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd404a350704e0483c7b2f08f07d77e, State: Initialized, Role: FOLLOWER
I20260927 03:23:20.514329 7758 consensus_queue.cc:260] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } }
I20260927 03:23:20.514564 7758 ts_tablet_manager.cc:1428] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 03:23:20.514575 7727 heartbeater.cc:503] Master 127.0.98.62:37939 was elected leader, sending a full tablet report...
I20260927 03:23:20.516542 7755 raft_consensus.cc:348] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } }
I20260927 03:23:20.516659 7755 raft_consensus.cc:374] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:20.516678 7755 raft_consensus.cc:729] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef4d9fd028fb4283b48cc865333cb3ca, State: Initialized, Role: FOLLOWER
I20260927 03:23:20.516763 7755 consensus_queue.cc:260] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } }
I20260927 03:23:20.516963 7755 ts_tablet_manager.cc:1428] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:23:20.517012 7591 heartbeater.cc:503] Master 127.0.98.62:37939 was elected leader, sending a full tablet report...
I20260927 03:23:20.524356 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.521931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.0.98.60:33087)
0927 03:23:20.521984 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.521986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.521994 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:20.522016 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.522017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.522036 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.522093 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:20.522205 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.522694 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.523631 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.524053 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.524113 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.524188 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.524244 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.524285 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:23:20.524356 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.521853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33087 (local address 127.0.0.1:45812)
0927 03:23:20.521894 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:20.521905 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:20.521938 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:20.522101 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:20.522102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:20.522103 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:20.522154 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.522155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.522762 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.522763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.523556 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.523558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.524122 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.524123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.524182 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.524186 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:20.524237 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:23:20.524285 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 03:23:20.524718 7283 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260927 03:23:20.525569 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4216573a82c74be9b92dbe0e0046c1ba. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } } }
I20260927 03:23:20.525724 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [sys.catalog]: This master's current role is: LEARNER
W20260927 03:23:20.571979 7593 tablet.cc:2369] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:23:20.577201 7763 raft_consensus.cc:482] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:20.577325 7763 raft_consensus.cc:504] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } }
I20260927 03:23:20.577690 7763 leader_election.cc:288] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 387121f928344d86a04295c84ae74dd7 (127.0.98.1:42473), bbd404a350704e0483c7b2f08f07d77e (127.0.98.3:45035)
I20260927 03:23:20.580569 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.578009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42631 (local address 127.0.98.1:42473)
0927 03:23:20.578078 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.578080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.578089 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:23:20.578115 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.578115 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.578124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.578191 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:20.578333 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.578911 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.579863 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.580239 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.580291 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.580301 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.580437 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.580474 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:23:20.580567 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.577889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42473 (local address 127.0.98.2:42631)
0927 03:23:20.577932 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:20.577943 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:20.577986 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:20.578207 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:20.578207 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:20.578209 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:20.578270 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.578271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.578978 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.578978 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.579784 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.579786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.580309 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.580310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.580380 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.580383 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:20.580423 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 03:23:20.580470 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":2}
I20260927 03:23:20.580622 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.578017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44259 (local address 127.0.98.3:45035)
0927 03:23:20.578080 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.578082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.578090 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:20.578113 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.578114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.578123 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.578161 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:20.578333 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.578790 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.579908 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.580261 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.580325 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.580335 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.580517 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.580563 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:23:20.580587 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.577871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45035 (local address 127.0.98.2:44259)
0927 03:23:20.577932 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:20.577950 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:20.577986 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:20.578189 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:20.578190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:20.578192 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:20.578263 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.578265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.578882 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.578883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.579820 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:20.579822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.580345 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.580346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:20.580414 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.580417 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:20.580468 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:23:20.580501 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":65}
I20260927 03:23:20.581133 7680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164f1fcd75234d0496a466ac73021b62" candidate_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd404a350704e0483c7b2f08f07d77e" is_pre_election: true
I20260927 03:23:20.581187 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164f1fcd75234d0496a466ac73021b62" candidate_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "387121f928344d86a04295c84ae74dd7" is_pre_election: true
I20260927 03:23:20.581290 7680 raft_consensus.cc:2454] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef4d9fd028fb4283b48cc865333cb3ca in term 0.
I20260927 03:23:20.581331 7408 raft_consensus.cc:2454] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef4d9fd028fb4283b48cc865333cb3ca in term 0.
I20260927 03:23:20.581521 7477 leader_election.cc:302] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 387121f928344d86a04295c84ae74dd7, bbd404a350704e0483c7b2f08f07d77e, ef4d9fd028fb4283b48cc865333cb3ca; no voters:
I20260927 03:23:20.581667 7763 raft_consensus.cc:2781] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:23:20.581738 7763 raft_consensus.cc:482] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:20.581755 7763 raft_consensus.cc:3037] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:20.582888 7763 raft_consensus.cc:504] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } }
I20260927 03:23:20.583012 7763 leader_election.cc:288] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [CANDIDATE]: Term 1 election: Requested vote from peers 387121f928344d86a04295c84ae74dd7 (127.0.98.1:42473), bbd404a350704e0483c7b2f08f07d77e (127.0.98.3:45035)
I20260927 03:23:20.583180 7680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164f1fcd75234d0496a466ac73021b62" candidate_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd404a350704e0483c7b2f08f07d77e"
I20260927 03:23:20.583218 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164f1fcd75234d0496a466ac73021b62" candidate_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "387121f928344d86a04295c84ae74dd7"
I20260927 03:23:20.583262 7680 raft_consensus.cc:3037] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:20.583320 7408 raft_consensus.cc:3037] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:20.584117 7408 raft_consensus.cc:2454] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef4d9fd028fb4283b48cc865333cb3ca in term 1.
I20260927 03:23:20.584271 7680 raft_consensus.cc:2454] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef4d9fd028fb4283b48cc865333cb3ca in term 1.
I20260927 03:23:20.584299 7478 leader_election.cc:302] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387121f928344d86a04295c84ae74dd7, ef4d9fd028fb4283b48cc865333cb3ca; no voters:
I20260927 03:23:20.584376 7763 raft_consensus.cc:2781] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:20.584510 7763 raft_consensus.cc:686] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [term 1 LEADER]: Becoming Leader. State: Replica: ef4d9fd028fb4283b48cc865333cb3ca, State: Running, Role: LEADER
I20260927 03:23:20.584600 7763 consensus_queue.cc:237] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } }
I20260927 03:23:20.585386 6489 catalog_manager.cc:5495] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca reported cstate change: term changed from 0 to 1, leader changed from <none> to ef4d9fd028fb4283b48cc865333cb3ca (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } health_report { overall_health: UNKNOWN } } }
I20260927 03:23:20.586076 7771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: true } } }
I20260927 03:23:20.586181 7771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:20.586738 7771 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba: attempting to promote NON_VOTER 2c20d31f9fb64676a11df189db754b54 to VOTER
W20260927 03:23:20.586931 7744 consensus_queue.cc:1159] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "af86687a976b42eeb8255db2ef798344" responder_term: 1 status { last_received { term: 1 index: 10 } last_committed_idx: 9 last_received_current_leader { term: 1 index: 10 } }
I20260927 03:23:20.587050 7771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } }
I20260927 03:23:20.587492 7283 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 1 LEARNER]: Refusing update from remote peer 4216573a82c74be9b92dbe0e0046c1ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 03:23:20.587530 7098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 FOLLOWER]: Refusing update from remote peer 4216573a82c74be9b92dbe0e0046c1ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 03:23:20.587687 7744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260927 03:23:20.587778 7771 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260927 03:23:20.588428 7744 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 2c20d31f9fb64676a11df189db754b54 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } } }
I20260927 03:23:20.588563 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 2c20d31f9fb64676a11df189db754b54 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } } }
I20260927 03:23:20.588723 7283 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 2c20d31f9fb64676a11df189db754b54 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } } }
I20260927 03:23:20.589339 7744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } } }
I20260927 03:23:20.589421 7744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:20.589468 7744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } } }
I20260927 03:23:20.589509 7744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:20.589556 7744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } } }
I20260927 03:23:20.589600 7744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:20.589710 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } } }
I20260927 03:23:20.589782 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:23:20.589960 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4216573a82c74be9b92dbe0e0046c1ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } } }
I20260927 03:23:20.590061 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:23:20.616899 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.614522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.0.98.2:36333)
0927 03:23:20.614572 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.614574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.614583 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:23:20.614623 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.614623 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.614643 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.614714 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:20.614847 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.615407 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.616141 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.616318 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.616365 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.616538 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:20.616644 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:20.616720 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.616806 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.616829 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:23:20.618606 7408 raft_consensus.cc:1261] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7 [term 1 FOLLOWER]: Refusing update from remote peer ef4d9fd028fb4283b48cc865333cb3ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:20.618635 7680 raft_consensus.cc:1261] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e [term 1 FOLLOWER]: Refusing update from remote peer ef4d9fd028fb4283b48cc865333cb3ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:20.618969 7769 consensus_queue.cc:1035] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:20.619093 7763 consensus_queue.cc:1035] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:20.621927 7775 mvcc.cc:205] Tried to move back new op lower bound from 7333803624931409920 to 7333803624795164672. Current Snapshot: MvccSnapshot[applied={T|T < 7333803624931409920}]
I20260927 03:23:20.623167 7777 mvcc.cc:205] Tried to move back new op lower bound from 7333803624931409920 to 7333803624795164672. Current Snapshot: MvccSnapshot[applied={T|T < 7333803624931409920 or (T in {7333803624931409920})}]
I20260927 03:23:20.623428 7776 mvcc.cc:205] Tried to move back new op lower bound from 7333803624931409920 to 7333803624795164672. Current Snapshot: MvccSnapshot[applied={T|T < 7333803624931409920}]
I20260927 03:23:20.624235 6488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:20.624327 6488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:20.625528 6488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4216573a82c74be9b92dbe0e0046c1ba: Sending DeleteTablet for 3 replicas of tablet 164f1fcd75234d0496a466ac73021b62
I20260927 03:23:20.625818 7660 tablet_service.cc:1514] Processing DeleteTablet for tablet 164f1fcd75234d0496a466ac73021b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34048
I20260927 03:23:20.625985 7524 tablet_service.cc:1514] Processing DeleteTablet for tablet 164f1fcd75234d0496a466ac73021b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33156
I20260927 03:23:20.626020 7388 tablet_service.cc:1514] Processing DeleteTablet for tablet 164f1fcd75234d0496a466ac73021b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59198
I20260927 03:23:20.626192 7786 tablet_replica.cc:330] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca: stopping tablet replica
I20260927 03:23:20.626322 7786 raft_consensus.cc:2229] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:23:20.626473 7786 raft_consensus.cc:2258] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:20.626755 7785 tablet_replica.cc:330] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e: stopping tablet replica
I20260927 03:23:20.626787 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 6457
I20260927 03:23:20.627318 7786 ts_tablet_manager.cc:1905] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:20.627835 7785 raft_consensus.cc:2229] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:20.627951 7785 raft_consensus.cc:2258] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:20.628356 7785 ts_tablet_manager.cc:1905] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:20.628788 7786 ts_tablet_manager.cc:1918] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:20.628849 7786 log.cc:1196] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/164f1fcd75234d0496a466ac73021b62
I20260927 03:23:20.629084 7786 ts_tablet_manager.cc:1939] T 164f1fcd75234d0496a466ac73021b62 P ef4d9fd028fb4283b48cc865333cb3ca: Deleting consensus metadata
I20260927 03:23:20.629733 7785 ts_tablet_manager.cc:1918] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:20.629808 7785 log.cc:1196] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/164f1fcd75234d0496a466ac73021b62
I20260927 03:23:20.630033 7785 ts_tablet_manager.cc:1939] T 164f1fcd75234d0496a466ac73021b62 P bbd404a350704e0483c7b2f08f07d77e: Deleting consensus metadata
W20260927 03:23:20.632030 7613 connection.cc:535] server connection from 127.0.0.1:34048 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 03:23:20.632071 7477 connection.cc:535] server connection from 127.0.0.1:33156 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:23:20.634953 7787 tablet_replica.cc:330] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7: stopping tablet replica
I20260927 03:23:20.635067 7787 raft_consensus.cc:2229] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:20.635171 7787 raft_consensus.cc:2258] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:20.635751 7787 ts_tablet_manager.cc:1905] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:20.637377 7787 ts_tablet_manager.cc:1918] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:20.637451 7787 log.cc:1196] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-0/wal/wals/164f1fcd75234d0496a466ac73021b62
I20260927 03:23:20.637712 7787 ts_tablet_manager.cc:1939] T 164f1fcd75234d0496a466ac73021b62 P 387121f928344d86a04295c84ae74dd7: Deleting consensus metadata
W20260927 03:23:20.638337 7343 connection.cc:441] server connection from 127.0.0.1:59198 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59198 (request call id 1) could send its response
I20260927 03:23:20.641009 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.633534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.0.98.61:33793)
0927 03:23:20.633579 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.633580 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.633908 (+ 328us) server_negotiation.cc:407] Connection header received
0927 03:23:20.634002 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.634003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.634017 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.634062 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:20.634701 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.635319 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.636280 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.636579 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.636622 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.638646 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:20.638647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:20.638648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:20.638655 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:20.640032 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.640327 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.640328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.640328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.640380 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.640534 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.640535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.640535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.640647 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:20.640729 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.640839 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.640917 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:23:20.641512 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:20.633533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45818 (local address 127.0.98.60:33087)
0927 03:23:20.633574 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:20.633576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:20.633884 (+ 308us) server_negotiation.cc:407] Connection header received
0927 03:23:20.633960 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:20.633961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:20.633972 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:20.634028 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:20.634862 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.635557 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.636704 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:20.637008 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:20.637050 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:20.638671 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:20.638673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:20.638673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:20.638680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:20.640286 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.640655 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.640656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.640656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.640708 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:20.640888 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:20.640889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:20.640889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:20.641025 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:20.641090 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:20.641379 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 03:23:20.641453 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:23:21.623426 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:21.623140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.2:34733)
0927 03:23:21.623263 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:21.623370 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260927 03:23:21.623651 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:21.623432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.3:38125)
0927 03:23:21.623534 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:21.623600 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
W20260927 03:23:21.623677 7591 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
W20260927 03:23:21.623822 7727 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
I20260927 03:23:21.624814 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:21.624315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.1:38609)
0927 03:23:21.624690 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:21.624759 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":59,"threads_started":1}
W20260927 03:23:21.625010 7455 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
I20260927 03:23:22.126343 7798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4216573a82c74be9b92dbe0e0046c1ba)
I20260927 03:23:22.126466 7798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } }
I20260927 03:23:22.126696 7799 raft_consensus.cc:482] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4216573a82c74be9b92dbe0e0046c1ba)
I20260927 03:23:22.126736 7798 leader_election.cc:288] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4216573a82c74be9b92dbe0e0046c1ba (127.0.98.62:37939), af86687a976b42eeb8255db2ef798344 (127.0.98.61:33793)
I20260927 03:23:22.126883 7799 raft_consensus.cc:504] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } }
I20260927 03:23:22.127097 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:22.126903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:44306)
0927 03:23:22.127006 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:22.127056 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":45,"threads_started":1}
W20260927 03:23:22.127206 7259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
I20260927 03:23:22.127556 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:22.127482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:44328)
0927 03:23:22.127511 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:22.127532 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260927 03:23:22.127640 7259 leader_election.cc:334] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4216573a82c74be9b92dbe0e0046c1ba (127.0.98.62:37939): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
I20260927 03:23:22.127801 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:22.127358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:44320)
0927 03:23:22.127457 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:22.127706 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":54,"threads_started":1}
I20260927 03:23:22.127810 7799 leader_election.cc:288] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4216573a82c74be9b92dbe0e0046c1ba (127.0.98.62:37939), 2c20d31f9fb64676a11df189db754b54 (127.0.98.60:33087)
W20260927 03:23:22.127918 7074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
I20260927 03:23:22.128381 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:22.128261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:44342)
0927 03:23:22.128329 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:22.128356 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 03:23:22.128492 7074 leader_election.cc:334] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4216573a82c74be9b92dbe0e0046c1ba (127.0.98.62:37939): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
I20260927 03:23:22.130398 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:22.126910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33793 (local address 127.0.0.1:55964)
0927 03:23:22.127048 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:22.127059 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:22.127123 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:22.127342 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:22.127344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:22.127346 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:22.127408 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:22.127411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:22.128515 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:22.128516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:22.129625 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:22.129628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:22.130222 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:22.130223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:22.130268 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:22.130271 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:22.130320 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:23:22.130353 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":45,"threads_started":1}
I20260927 03:23:22.130422 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:22.126948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.0.98.61:33793)
0927 03:23:22.127054 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:22.127056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:22.127200 (+ 144us) server_negotiation.cc:407] Connection header received
0927 03:23:22.127238 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:22.127239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:22.127250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:22.127313 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:22.127490 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:22.128433 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:22.129713 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:22.130151 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:22.130245 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:22.130256 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:22.130329 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 03:23:22.130363 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":59,"threads_started":1}
I20260927 03:23:22.130774 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:22.127716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.0.98.60:33087)
0927 03:23:22.127842 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:22.127843 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:22.127898 (+ 55us) server_negotiation.cc:407] Connection header received
0927 03:23:22.127919 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:22.127919 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:22.127928 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:22.128066 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:22.128192 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:22.128804 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:22.129984 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:22.130494 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:22.130561 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:22.130570 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:22.130679 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 03:23:22.130713 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
I20260927 03:23:22.130770 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:22.127549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33087 (local address 127.0.0.1:45830)
0927 03:23:22.127674 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:22.127697 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:22.127747 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:22.128076 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:22.128077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:22.128079 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:22.128117 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:22.128118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:22.128876 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:22.128877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:22.129913 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:22.129915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:22.130563 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:22.130564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:22.130623 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:22.130626 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:22.130670 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:23:22.130709 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":48,"threads_started":1}
I20260927 03:23:22.130760 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c20d31f9fb64676a11df189db754b54" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "af86687a976b42eeb8255db2ef798344" is_pre_election: true
I20260927 03:23:22.130877 7098 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c20d31f9fb64676a11df189db754b54 in term 1.
I20260927 03:23:22.131040 7260 leader_election.cc:302] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c20d31f9fb64676a11df189db754b54, af86687a976b42eeb8255db2ef798344; no voters: 4216573a82c74be9b92dbe0e0046c1ba
I20260927 03:23:22.131034 7283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af86687a976b42eeb8255db2ef798344" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c20d31f9fb64676a11df189db754b54" is_pre_election: true
I20260927 03:23:22.131124 7283 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af86687a976b42eeb8255db2ef798344 in term 1.
I20260927 03:23:22.131166 7798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 03:23:22.131198 7798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4216573a82c74be9b92dbe0e0046c1ba)
I20260927 03:23:22.131215 7798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:23:22.131253 7072 leader_election.cc:302] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c20d31f9fb64676a11df189db754b54, af86687a976b42eeb8255db2ef798344; no voters: 4216573a82c74be9b92dbe0e0046c1ba
I20260927 03:23:22.131338 7799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 03:23:22.131371 7799 raft_consensus.cc:482] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4216573a82c74be9b92dbe0e0046c1ba)
I20260927 03:23:22.131387 7799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:23:22.132076 7799 raft_consensus.cc:504] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } }
I20260927 03:23:22.132076 7798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } }
I20260927 03:23:22.132200 7798 leader_election.cc:288] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [CANDIDATE]: Term 2 election: Requested vote from peers 4216573a82c74be9b92dbe0e0046c1ba (127.0.98.62:37939), af86687a976b42eeb8255db2ef798344 (127.0.98.61:33793)
I20260927 03:23:22.132408 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:22.132275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:44358)
0927 03:23:22.132363 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:22.132383 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 03:23:22.132396 7283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af86687a976b42eeb8255db2ef798344" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c20d31f9fb64676a11df189db754b54"
I20260927 03:23:22.132335 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c20d31f9fb64676a11df189db754b54" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "af86687a976b42eeb8255db2ef798344"
I20260927 03:23:22.132516 7283 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate af86687a976b42eeb8255db2ef798344 in current term 2: Already voted for candidate 2c20d31f9fb64676a11df189db754b54 in this term.
I20260927 03:23:22.132546 7098 raft_consensus.cc:2379] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2c20d31f9fb64676a11df189db754b54 in current term 2: Already voted for candidate af86687a976b42eeb8255db2ef798344 in this term.
I20260927 03:23:22.132645 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:22.132275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:44362)
0927 03:23:22.132312 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:22.132334 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 03:23:22.132194 7799 leader_election.cc:288] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [CANDIDATE]: Term 2 election: Requested vote from peers 4216573a82c74be9b92dbe0e0046c1ba (127.0.98.62:37939), 2c20d31f9fb64676a11df189db754b54 (127.0.98.60:33087)
I20260927 03:23:22.132905 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:22.132804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:44372)
0927 03:23:22.132863 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:22.132886 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260927 03:23:22.132963 7259 leader_election.cc:334] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4216573a82c74be9b92dbe0e0046c1ba (127.0.98.62:37939): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
I20260927 03:23:22.132988 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:22.132903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:44378)
0927 03:23:22.132943 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:22.132969 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 03:23:22.133038 7259 leader_election.cc:302] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c20d31f9fb64676a11df189db754b54; no voters: 4216573a82c74be9b92dbe0e0046c1ba, af86687a976b42eeb8255db2ef798344
W20260927 03:23:22.133044 7074 leader_election.cc:334] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4216573a82c74be9b92dbe0e0046c1ba (127.0.98.62:37939): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
I20260927 03:23:22.133090 7074 leader_election.cc:302] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af86687a976b42eeb8255db2ef798344; no voters: 2c20d31f9fb64676a11df189db754b54, 4216573a82c74be9b92dbe0e0046c1ba
I20260927 03:23:22.133098 7798 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 03:23:22.133148 7799 raft_consensus.cc:2726] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 03:23:22.624281 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:22.624028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.2:33399)
0927 03:23:22.624156 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:22.624209 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260927 03:23:22.624329 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:22.624140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.3:51089)
0927 03:23:22.624251 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:22.624288 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260927 03:23:22.625408 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:22.625255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.1:59677)
0927 03:23:22.625343 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:22.625374 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":48,"threads_started":1}
I20260927 03:23:23.624922 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.624730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.3:34801)
0927 03:23:23.624837 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.624875 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":58,"threads_started":1}
I20260927 03:23:23.624929 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.624731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.2:56025)
0927 03:23:23.624857 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.624888 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260927 03:23:23.625921 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.625739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.1:56031)
0927 03:23:23.625838 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.625877 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
I20260927 03:23:23.817050 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:23.817148 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } }
I20260927 03:23:23.817286 7814 leader_election.cc:288] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4216573a82c74be9b92dbe0e0046c1ba (127.0.98.62:37939), 2c20d31f9fb64676a11df189db754b54 (127.0.98.60:33087)
I20260927 03:23:23.817538 7283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af86687a976b42eeb8255db2ef798344" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c20d31f9fb64676a11df189db754b54" is_pre_election: true
I20260927 03:23:23.817623 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.817427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:44590)
0927 03:23:23.817536 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.817582 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260927 03:23:23.817654 7283 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af86687a976b42eeb8255db2ef798344 in term 2.
I20260927 03:23:23.817798 7072 leader_election.cc:302] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c20d31f9fb64676a11df189db754b54, af86687a976b42eeb8255db2ef798344; no voters:
I20260927 03:23:23.817883 7814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 03:23:23.817921 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:23.817941 7814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 2 FOLLOWER]: Advancing to term 3
I20260927 03:23:23.818091 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.817979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:44592)
0927 03:23:23.818010 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.818037 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260927 03:23:23.818162 7074 leader_election.cc:334] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4216573a82c74be9b92dbe0e0046c1ba (127.0.98.62:37939): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
I20260927 03:23:23.818682 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } }
I20260927 03:23:23.818809 7814 leader_election.cc:288] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [CANDIDATE]: Term 3 election: Requested vote from peers 4216573a82c74be9b92dbe0e0046c1ba (127.0.98.62:37939), 2c20d31f9fb64676a11df189db754b54 (127.0.98.60:33087)
I20260927 03:23:23.819000 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.818906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:44602)
0927 03:23:23.818947 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.818975 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 03:23:23.818980 7283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af86687a976b42eeb8255db2ef798344" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c20d31f9fb64676a11df189db754b54"
I20260927 03:23:23.819058 7283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 2 FOLLOWER]: Advancing to term 3
I20260927 03:23:23.819422 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.819331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:44604)
0927 03:23:23.819371 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.819398 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 03:23:23.819491 7074 leader_election.cc:334] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4216573a82c74be9b92dbe0e0046c1ba (127.0.98.62:37939): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
I20260927 03:23:23.819867 7283 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af86687a976b42eeb8255db2ef798344 in term 3.
I20260927 03:23:23.820058 7072 leader_election.cc:302] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c20d31f9fb64676a11df189db754b54, af86687a976b42eeb8255db2ef798344; no voters: 4216573a82c74be9b92dbe0e0046c1ba
I20260927 03:23:23.820147 7814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 03:23:23.820276 7814 raft_consensus.cc:686] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [term 3 LEADER]: Becoming Leader. State: Replica: af86687a976b42eeb8255db2ef798344, State: Running, Role: LEADER
I20260927 03:23:23.820349 7814 consensus_queue.cc:237] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } }
I20260927 03:23:23.820832 7817 sys_catalog.cc:455] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af86687a976b42eeb8255db2ef798344. Latest consensus state: current_term: 3 leader_uuid: "af86687a976b42eeb8255db2ef798344" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } } }
I20260927 03:23:23.820957 7817 sys_catalog.cc:458] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:23.821151 7819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:23.821632 7819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:23.821698 7819 catalog_manager.cc:1259] Loaded cluster ID: 5476a318fb324d70a21b536467118dfc
I20260927 03:23:23.821713 7819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:23:23.821890 7819 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:23.821950 7819 catalog_manager.cc:5879] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: Loaded TSK: 0
I20260927 03:23:23.822072 7819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:23:23.880681 7087 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:23:23.882390 7087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:23:23.883441 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.883331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:44624)
0927 03:23:23.883380 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.883414 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 03:23:23.883448 7283 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [term 3 FOLLOWER]: Refusing update from remote peer af86687a976b42eeb8255db2ef798344: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 03:23:23.883816 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.883738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:44640)
0927 03:23:23.883767 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.883795 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 03:23:23.883775 7814 consensus_queue.cc:1035] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 03:23:23.883893 7074 consensus_peers.cc:487] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 -> Peer 4216573a82c74be9b92dbe0e0046c1ba (127.0.98.62:37939): Couldn't send request to peer 4216573a82c74be9b92dbe0e0046c1ba. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 03:23:23.884313 7821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af86687a976b42eeb8255db2ef798344. Latest consensus state: current_term: 3 leader_uuid: "af86687a976b42eeb8255db2ef798344" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } } }
I20260927 03:23:23.884405 7821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:23:23.884697 7820 mvcc.cc:205] Tried to move back new op lower bound from 7333803638304825344 to 7333803638049177600. Current Snapshot: MvccSnapshot[applied={T|T < 7333803638304825344}]
I20260927 03:23:23.884785 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "af86687a976b42eeb8255db2ef798344" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } } }
I20260927 03:23:23.884863 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344 [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:23.884984 7822 mvcc.cc:205] Tried to move back new op lower bound from 7333803638304825344 to 7333803638049177600. Current Snapshot: MvccSnapshot[applied={T|T < 7333803638304825344}]
I20260927 03:23:23.885143 7821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "af86687a976b42eeb8255db2ef798344" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4216573a82c74be9b92dbe0e0046c1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "af86687a976b42eeb8255db2ef798344" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33793 } attrs { promote: false } } peers { permanent_uuid: "2c20d31f9fb64676a11df189db754b54" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33087 } attrs { promote: false } } }
I20260927 03:23:23.885288 7821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c20d31f9fb64676a11df189db754b54 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:23:23.889642 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.886444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36333 (local address 127.0.0.1:33174)
0927 03:23:23.886737 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.886744 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:23.886775 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:23.886954 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:23.886955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:23.886957 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:23.887001 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:23.887003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.887991 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.887992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:23.888787 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:23.888789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.889400 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.889400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:23.889460 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:23.889464 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:23.889515 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:23:23.889559 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":31,"thread_start_us":89,"threads_started":1}
I20260927 03:23:23.889647 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.886708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33174 (local address 127.0.98.2:36333)
0927 03:23:23.886849 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:23.886851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:23.886858 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:23:23.886882 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:23.886883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:23.886893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:23.886948 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:23.887115 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.887896 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.888872 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.889324 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.889397 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:23.889410 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:23.889524 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 03:23:23.889564 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260927 03:23:23.889886 7524 tablet_service.cc:1467] Processing CreateTablet for tablet 5175ea6ea9fa4b6ca6d16453b4a3cbf6 (DEFAULT_TABLE table=test-table [id=f321b2531484407e9830f42d35d62984]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:23.890023 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5175ea6ea9fa4b6ca6d16453b4a3cbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:23.890043 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.886435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45035 (local address 127.0.0.1:34064)
0927 03:23:23.886567 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.886576 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:23.886664 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:23.887002 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:23.887003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:23.887005 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:23.887043 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:23.887044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.888002 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.888003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:23.889093 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:23.889095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.889814 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.889815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:23.889881 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:23.889885 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:23.889930 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:23:23.889972 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":22,"thread_start_us":93,"threads_started":1}
I20260927 03:23:23.890048 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.886630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34064 (local address 127.0.98.3:45035)
0927 03:23:23.886854 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:23.886856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:23.886863 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:23:23.886887 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:23.886887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:23.886896 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:23.886959 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:23.887152 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.887900 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.889171 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.889706 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.889771 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:23.889783 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:23.889945 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 03:23:23.889982 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":54,"threads_started":1}
I20260927 03:23:23.890895 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.886396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42473 (local address 127.0.0.1:59208)
0927 03:23:23.886447 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.886458 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:23.886507 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:23.886860 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:23.886861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:23.886863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:23.886950 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:23.886952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.887736 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.887737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:23.888803 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:23.888805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.890662 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.890663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:23.890727 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:23.890732 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:23:23.890763 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 03:23:23.890803 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:23:23.891379 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.886598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.0.98.1:42473)
0927 03:23:23.886696 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:23.886698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:23.886706 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:23.886731 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:23.886732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:23.886741 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:23.887071 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:23.887110 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.887661 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.889163 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.889664 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.891048 (+ 1384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:23.891073 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:23.891091 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 03:23:23.891136 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":53,"threads_started":1}
I20260927 03:23:23.891532 7831 tablet_bootstrap.cc:492] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca: Bootstrap starting.
I20260927 03:23:23.891590 7388 tablet_service.cc:1467] Processing CreateTablet for tablet 5175ea6ea9fa4b6ca6d16453b4a3cbf6 (DEFAULT_TABLE table=test-table [id=f321b2531484407e9830f42d35d62984]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:23.891692 7388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5175ea6ea9fa4b6ca6d16453b4a3cbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:23.891830 7660 tablet_service.cc:1467] Processing CreateTablet for tablet 5175ea6ea9fa4b6ca6d16453b4a3cbf6 (DEFAULT_TABLE table=test-table [id=f321b2531484407e9830f42d35d62984]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:23.891938 7660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5175ea6ea9fa4b6ca6d16453b4a3cbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:23.891937 7831 tablet_bootstrap.cc:654] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:23.892463 7831 tablet_bootstrap.cc:492] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca: No bootstrap required, opened a new log
I20260927 03:23:23.892510 7831 ts_tablet_manager.cc:1397] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:23.892623 7831 raft_consensus.cc:348] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260927 03:23:23.892712 7831 raft_consensus.cc:374] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:23.892733 7831 raft_consensus.cc:729] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef4d9fd028fb4283b48cc865333cb3ca, State: Initialized, Role: FOLLOWER
I20260927 03:23:23.892786 7831 consensus_queue.cc:260] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260927 03:23:23.892976 7831 ts_tablet_manager.cc:1428] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:23.893059 7834 tablet_bootstrap.cc:492] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7: Bootstrap starting.
I20260927 03:23:23.893523 7834 tablet_bootstrap.cc:654] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:23.893715 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.893376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.2:55495)
0927 03:23:23.893660 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.893692 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
I20260927 03:23:23.893888 7835 tablet_bootstrap.cc:492] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e: Bootstrap starting.
I20260927 03:23:23.894146 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.894049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.2:32951)
0927 03:23:23.894097 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.894121 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:23:23.894187 7834 tablet_bootstrap.cc:492] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7: No bootstrap required, opened a new log
I20260927 03:23:23.894223 7834 ts_tablet_manager.cc:1397] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:23.894356 7835 tablet_bootstrap.cc:654] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:23.894335 7834 raft_consensus.cc:348] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260927 03:23:23.894398 7834 raft_consensus.cc:374] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:23.894424 7834 raft_consensus.cc:729] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387121f928344d86a04295c84ae74dd7, State: Initialized, Role: FOLLOWER
I20260927 03:23:23.894490 7592 heartbeater.cc:503] Master 127.0.98.61:33793 was elected leader, sending a full tablet report...
I20260927 03:23:23.894491 7834 consensus_queue.cc:260] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260927 03:23:23.894663 7834 ts_tablet_manager.cc:1428] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:23.895134 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.895026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.1:46777)
0927 03:23:23.895070 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.895100 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 03:23:23.895295 7835 tablet_bootstrap.cc:492] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e: No bootstrap required, opened a new log
I20260927 03:23:23.895342 7835 ts_tablet_manager.cc:1397] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:23.895466 7835 raft_consensus.cc:348] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260927 03:23:23.895530 7835 raft_consensus.cc:374] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:23.895543 7835 raft_consensus.cc:729] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd404a350704e0483c7b2f08f07d77e, State: Initialized, Role: FOLLOWER
I20260927 03:23:23.895576 7835 consensus_queue.cc:260] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260927 03:23:23.895743 7835 ts_tablet_manager.cc:1428] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:23.895968 7456 heartbeater.cc:503] Master 127.0.98.61:33793 was elected leader, sending a full tablet report...
I20260927 03:23:23.896075 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.895939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.3:37407)
0927 03:23:23.896024 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.896050 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 03:23:23.896379 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.896303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.3:51127)
0927 03:23:23.896337 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.896359 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 03:23:23.896400 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.896321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.1:42681)
0927 03:23:23.896355 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.896379 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 03:23:23.918783 7838 raft_consensus.cc:482] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:23.918910 7838 raft_consensus.cc:504] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260927 03:23:23.919255 7838 leader_election.cc:288] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbd404a350704e0483c7b2f08f07d77e (127.0.98.3:45035), ef4d9fd028fb4283b48cc865333cb3ca (127.0.98.2:36333)
I20260927 03:23:23.922070 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.919413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45035 (local address 127.0.98.1:45353)
0927 03:23:23.919721 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.919736 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:23.919782 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:23.919881 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:23.919881 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:23.919883 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:23.919925 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:23.919927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.920514 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.920515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:23.921279 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:23.921281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.921828 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.921829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:23.921888 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:23.921892 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:23.921938 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:23:23.921979 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":156,"thread_start_us":92,"threads_started":1}
I20260927 03:23:23.922056 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.919472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45353 (local address 127.0.98.3:45035)
0927 03:23:23.919525 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:23.919527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:23.919789 (+ 262us) server_negotiation.cc:407] Connection header received
0927 03:23:23.919816 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:23.919817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:23.919829 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:23.919877 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:23.920000 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.920437 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.921363 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.921767 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.921825 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:23.921838 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:23.921950 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 03:23:23.921991 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:23:23.922209 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.919484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48235 (local address 127.0.98.2:36333)
0927 03:23:23.919525 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:23.919527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:23.919537 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:23.919571 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:23.919572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:23.919584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:23.919687 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:23.919856 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.920404 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.921369 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.921827 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.921892 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:23.921904 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:23.922103 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 03:23:23.922154 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:23:23.922184 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.919410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36333 (local address 127.0.98.1:48235)
0927 03:23:23.919461 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.919472 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:23.919526 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:23.919699 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:23.919700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:23.919703 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:23.919776 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:23.919779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.920485 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.920486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:23.921281 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:23.921283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.921949 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.921951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:23.922028 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:23.922032 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:23.922080 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:23:23.922119 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 03:23:23.922508 7544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5175ea6ea9fa4b6ca6d16453b4a3cbf6" candidate_uuid: "387121f928344d86a04295c84ae74dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" is_pre_election: true
I20260927 03:23:23.922612 7680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5175ea6ea9fa4b6ca6d16453b4a3cbf6" candidate_uuid: "387121f928344d86a04295c84ae74dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd404a350704e0483c7b2f08f07d77e" is_pre_election: true
I20260927 03:23:23.922662 7544 raft_consensus.cc:2454] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 387121f928344d86a04295c84ae74dd7 in term 0.
I20260927 03:23:23.922711 7680 raft_consensus.cc:2454] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 387121f928344d86a04295c84ae74dd7 in term 0.
I20260927 03:23:23.922883 7341 leader_election.cc:302] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387121f928344d86a04295c84ae74dd7, bbd404a350704e0483c7b2f08f07d77e; no voters:
I20260927 03:23:23.923039 7838 raft_consensus.cc:2781] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:23:23.923076 7838 raft_consensus.cc:482] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:23.923089 7838 raft_consensus.cc:3037] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:23.923713 7838 raft_consensus.cc:504] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260927 03:23:23.923835 7838 leader_election.cc:288] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [CANDIDATE]: Term 1 election: Requested vote from peers bbd404a350704e0483c7b2f08f07d77e (127.0.98.3:45035), ef4d9fd028fb4283b48cc865333cb3ca (127.0.98.2:36333)
I20260927 03:23:23.923979 7544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5175ea6ea9fa4b6ca6d16453b4a3cbf6" candidate_uuid: "387121f928344d86a04295c84ae74dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef4d9fd028fb4283b48cc865333cb3ca"
I20260927 03:23:23.923995 7680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5175ea6ea9fa4b6ca6d16453b4a3cbf6" candidate_uuid: "387121f928344d86a04295c84ae74dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd404a350704e0483c7b2f08f07d77e"
I20260927 03:23:23.924055 7544 raft_consensus.cc:3037] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:23.924057 7680 raft_consensus.cc:3037] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:23.924587 7680 raft_consensus.cc:2454] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 387121f928344d86a04295c84ae74dd7 in term 1.
I20260927 03:23:23.924746 7544 raft_consensus.cc:2454] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 387121f928344d86a04295c84ae74dd7 in term 1.
I20260927 03:23:23.924764 7341 leader_election.cc:302] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387121f928344d86a04295c84ae74dd7, bbd404a350704e0483c7b2f08f07d77e; no voters:
I20260927 03:23:23.924840 7838 raft_consensus.cc:2781] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:23.924985 7838 raft_consensus.cc:686] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [term 1 LEADER]: Becoming Leader. State: Replica: 387121f928344d86a04295c84ae74dd7, State: Running, Role: LEADER
I20260927 03:23:23.925107 7838 consensus_queue.cc:237] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260927 03:23:23.925297 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.925195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.1:51605)
0927 03:23:23.925229 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.925266 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 03:23:23.925927 7087 catalog_manager.cc:5495] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 387121f928344d86a04295c84ae74dd7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "387121f928344d86a04295c84ae74dd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "387121f928344d86a04295c84ae74dd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } health_report { overall_health: UNKNOWN } } }
W20260927 03:23:23.933851 7457 tablet.cc:2369] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:23:23.935601 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.933342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59214 (local address 127.0.98.1:42473)
0927 03:23:23.933389 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:23.933391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:23.933398 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:23:23.933436 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:23.933437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:23.933455 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:23.933516 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:23.933677 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.934187 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.934953 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:23.935162 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:23.935214 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:23.935306 (+ 92us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:23.935426 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:23.935472 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:23.935508 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 03:23:23.935525 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:23:23.936740 7680 raft_consensus.cc:1261] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e [term 1 FOLLOWER]: Refusing update from remote peer 387121f928344d86a04295c84ae74dd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:23.937022 7838 consensus_queue.cc:1035] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbd404a350704e0483c7b2f08f07d77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:23.937390 7544 raft_consensus.cc:1261] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca [term 1 FOLLOWER]: Refusing update from remote peer 387121f928344d86a04295c84ae74dd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:23.937526 7728 heartbeater.cc:503] Master 127.0.98.61:33793 was elected leader, sending a full tablet report...
I20260927 03:23:23.937739 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.937628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.3:56257)
0927 03:23:23.937674 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.937707 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 03:23:23.937778 7838 consensus_queue.cc:1035] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P 387121f928344d86a04295c84ae74dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef4d9fd028fb4283b48cc865333cb3ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:23.938491 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.938394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.2:39679)
0927 03:23:23.938435 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.938460 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 03:23:23.939041 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.938928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.1:34249)
0927 03:23:23.938990 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.939013 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 03:23:23.939608 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.939521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.1:45789)
0927 03:23:23.939564 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.939586 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 03:23:23.939961 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.939875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.3:40321)
0927 03:23:23.939910 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.939932 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 03:23:23.940230 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.940067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.2:50953)
0927 03:23:23.940174 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.940202 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 03:23:23.941695 7087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:23.941792 7087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:23.942961 7087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af86687a976b42eeb8255db2ef798344: Sending DeleteTablet for 3 replicas of tablet 5175ea6ea9fa4b6ca6d16453b4a3cbf6
I20260927 03:23:23.943259 7524 tablet_service.cc:1514] Processing DeleteTablet for tablet 5175ea6ea9fa4b6ca6d16453b4a3cbf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:23 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33174
I20260927 03:23:23.943316 7660 tablet_service.cc:1514] Processing DeleteTablet for tablet 5175ea6ea9fa4b6ca6d16453b4a3cbf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:23 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34064
I20260927 03:23:23.943570 7388 tablet_service.cc:1514] Processing DeleteTablet for tablet 5175ea6ea9fa4b6ca6d16453b4a3cbf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:23 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59208
I20260927 03:23:23.943578 7854 tablet_replica.cc:330] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca: stopping tablet replica
I20260927 03:23:23.943841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 7325
I20260927 03:23:23.944489 7855 tablet_replica.cc:330] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e: stopping tablet replica
I20260927 03:23:23.944548 7855 raft_consensus.cc:2229] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:23.944598 7855 raft_consensus.cc:2258] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:23.944919 7855 ts_tablet_manager.cc:1905] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:23.946177 7854 raft_consensus.cc:2229] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:23.946250 7854 raft_consensus.cc:2258] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:23.946273 7855 ts_tablet_manager.cc:1918] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:23.946331 7855 log.cc:1196] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/5175ea6ea9fa4b6ca6d16453b4a3cbf6
I20260927 03:23:23.946539 7855 ts_tablet_manager.cc:1939] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P bbd404a350704e0483c7b2f08f07d77e: Deleting consensus metadata
I20260927 03:23:23.946586 7854 ts_tablet_manager.cc:1905] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:23.946835 7072 catalog_manager.cc:4841] TS bbd404a350704e0483c7b2f08f07d77e (127.0.98.3:45035): tablet 5175ea6ea9fa4b6ca6d16453b4a3cbf6 (table test-table [id=f321b2531484407e9830f42d35d62984]) successfully deleted
I20260927 03:23:23.948103 7854 ts_tablet_manager.cc:1918] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:23.948163 7854 log.cc:1196] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/5175ea6ea9fa4b6ca6d16453b4a3cbf6
I20260927 03:23:23.948369 7854 ts_tablet_manager.cc:1939] T 5175ea6ea9fa4b6ca6d16453b4a3cbf6 P ef4d9fd028fb4283b48cc865333cb3ca: Deleting consensus metadata
I20260927 03:23:23.948967 7074 catalog_manager.cc:4841] TS ef4d9fd028fb4283b48cc865333cb3ca (127.0.98.2:36333): tablet 5175ea6ea9fa4b6ca6d16453b4a3cbf6 (table test-table [id=f321b2531484407e9830f42d35d62984]) successfully deleted
I20260927 03:23:23.950029 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:23.949937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42473 (local address 127.0.0.1:59216)
0927 03:23:23.949973 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:23.950001 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 03:23:23.950179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 7461
W20260927 03:23:23.950186 7073 catalog_manager.cc:4584] TS 387121f928344d86a04295c84ae74dd7 (127.0.98.1:42473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5175ea6ea9fa4b6ca6d16453b4a3cbf6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42473: connect: Connection refused (error 111)
I20260927 03:23:23.955135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 7597
I20260927 03:23:23.959093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 6953
I20260927 03:23:23.963372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 7142
2026-09-27T03:23:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6228 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:23:24 dist-test-slave-gw2v krb5kdc[7859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:23:24 dist-test-slave-gw2v krb5kdc[7859](info): set up 2 sockets
Sep 27 03:23:24 dist-test-slave-gw2v krb5kdc[7859](info): commencing operation
krb5kdc: starting...
W20260927 03:23:24.907209 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:23:24 dist-test-slave-gw2v krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:23:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 03:23:25.024026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39957
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:25.107089 7872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:25.107307 7872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:25.107331 7872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:25.108554 7872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:25.108599 7872 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 03:23:25.108611 7872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:25.108623 7872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:25.108634 7872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:25.108644 7872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:25.110050 7872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39957
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:25.110245 7872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:25.110416 7872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:23:25.115142 7872 server_base.cc:1034] running on GCE node
W20260927 03:23:25.115118 7877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:25.115108 7878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:25.115123 7880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:25.115607 7872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:25.115908 7872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:25.117074 7872 hybrid_clock.cc:648] HybridClock initialized: now 1790479405117060 us; error 31 us; skew 500 ppm
Sep 27 03:23:25 dist-test-slave-gw2v krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479405, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:25.119781 7872 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 03:23:25.120205 7872 webserver.cc:460] Webserver started at http://127.0.98.62:46659/ using document root <none> and password file <none>
I20260927 03:23:25.120400 7872 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:25.120441 7872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:25.120529 7872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:25.121549 7872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "663c2a7162da4463b85aa8e2064b2f8a"
format_stamp: "Formatted at 2026-09-27 03:23:25 on dist-test-slave-gw2v"
server_key: "8e5a749208e042c60f6eed420aab7897"
server_key_iv: "f413665673f664a7f7a3ba690436ee49"
server_key_version: "encryptionkey@0"
I20260927 03:23:25.121881 7872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "663c2a7162da4463b85aa8e2064b2f8a"
format_stamp: "Formatted at 2026-09-27 03:23:25 on dist-test-slave-gw2v"
server_key: "8e5a749208e042c60f6eed420aab7897"
server_key_iv: "f413665673f664a7f7a3ba690436ee49"
server_key_version: "encryptionkey@0"
I20260927 03:23:25.123144 7872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:23:25.123886 7887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.124058 7872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:25.124133 7872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "663c2a7162da4463b85aa8e2064b2f8a"
format_stamp: "Formatted at 2026-09-27 03:23:25 on dist-test-slave-gw2v"
server_key: "8e5a749208e042c60f6eed420aab7897"
server_key_iv: "f413665673f664a7f7a3ba690436ee49"
server_key_version: "encryptionkey@0"
I20260927 03:23:25.124205 7872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:25.133857 7872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:25.135614 7872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:25.135728 7872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:25.139916 7872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39957
I20260927 03:23:25.139994 7949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39957 every 8 connection(s)
I20260927 03:23:25.140275 7872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:25.140885 7950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:25.143112 7950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a: Bootstrap starting.
I20260927 03:23:25.143631 7950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:25.143850 7950 log.cc:824] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:25.144510 7950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a: No bootstrap required, opened a new log
I20260927 03:23:25.146471 7950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663c2a7162da4463b85aa8e2064b2f8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39957 } }
I20260927 03:23:25.146605 7950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:25.146625 7950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663c2a7162da4463b85aa8e2064b2f8a, State: Initialized, Role: FOLLOWER
I20260927 03:23:25.146733 7950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663c2a7162da4463b85aa8e2064b2f8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39957 } }
I20260927 03:23:25.146788 7950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:25.146837 7950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:25.146880 7950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:25.147372 7950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663c2a7162da4463b85aa8e2064b2f8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39957 } }
I20260927 03:23:25.147477 7950 leader_election.cc:302] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 663c2a7162da4463b85aa8e2064b2f8a; no voters:
I20260927 03:23:25.147624 7950 leader_election.cc:288] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:25.147677 7953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:25.147861 7950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:25.147874 7953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [term 1 LEADER]: Becoming Leader. State: Replica: 663c2a7162da4463b85aa8e2064b2f8a, State: Running, Role: LEADER
I20260927 03:23:25.148012 7953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663c2a7162da4463b85aa8e2064b2f8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39957 } }
I20260927 03:23:25.148365 7955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 663c2a7162da4463b85aa8e2064b2f8a. Latest consensus state: current_term: 1 leader_uuid: "663c2a7162da4463b85aa8e2064b2f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663c2a7162da4463b85aa8e2064b2f8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39957 } } }
I20260927 03:23:25.148386 7954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "663c2a7162da4463b85aa8e2064b2f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663c2a7162da4463b85aa8e2064b2f8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39957 } } }
I20260927 03:23:25.148499 7954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:25.148693 7955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:25.149070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 7872
I20260927 03:23:25.149160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:25.149348 392 external_mini_cluster.cc:1428] Setting key a4705eb822ca68ec2544c768208152bd
I20260927 03:23:25.148703 7962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:25.149986 7962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:25.151473 7962 catalog_manager.cc:1347] Generated new cluster ID: 88731ce60c604e968aa2b0207b30101d
I20260927 03:23:25.151525 7962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:23:25 dist-test-slave-gw2v krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:23:25.161665 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.149829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.0.98.62:39957)
0927 03:23:25.150771 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.150780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.150808 (+ 28us) server_negotiation.cc:407] Connection header received
0927 03:23:25.150949 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.150955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.151005 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.151152 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:25.151983 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.153407 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.154509 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.155247 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.155366 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.158063 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:25.158068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:25.158068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:25.158082 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:25.160006 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.160354 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.160357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.160357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.160471 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.160636 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.160637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.160637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.160945 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:25.161108 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.161173 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.161410 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":884,"thread_start_us":70,"threads_started":1}
I20260927 03:23:25.196861 7962 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:25.197453 7962 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:25.201216 7962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a: Generated new TSK 0
I20260927 03:23:25.201397 7962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 03:23:25.231623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39957
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:25.313103 7976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:25.313330 7976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:25.313365 7976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:25.314579 7976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:25.314627 7976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:25.314654 7976 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:23:25.314678 7976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:23:25.316321 7976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:25.316591 7976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:25.316809 7976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:25.322355 7982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:25.322427 7984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:25.322454 7981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:25.322719 7976 server_base.cc:1034] running on GCE node
I20260927 03:23:25.322959 7976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:25.323246 7976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:25.324453 7976 hybrid_clock.cc:648] HybridClock initialized: now 1790479405324467 us; error 60 us; skew 500 ppm
Sep 27 03:23:25 dist-test-slave-gw2v krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479405, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:25.327129 7976 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 03:23:25.327662 7976 webserver.cc:460] Webserver started at http://127.0.98.1:42015/ using document root <none> and password file <none>
I20260927 03:23:25.327873 7976 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:25.327973 7976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:25.328094 7976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:25.329151 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09"
format_stamp: "Formatted at 2026-09-27 03:23:25 on dist-test-slave-gw2v"
server_key: "f5a141fb3f18214261bae844457f28d1"
server_key_iv: "c42ea6fbc69f075ddf0377f112530ce2"
server_key_version: "encryptionkey@0"
I20260927 03:23:25.329469 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09"
format_stamp: "Formatted at 2026-09-27 03:23:25 on dist-test-slave-gw2v"
server_key: "f5a141fb3f18214261bae844457f28d1"
server_key_iv: "c42ea6fbc69f075ddf0377f112530ce2"
server_key_version: "encryptionkey@0"
I20260927 03:23:25.330789 7976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:23:25.331601 7991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.331759 7976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:25.331857 7976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09"
format_stamp: "Formatted at 2026-09-27 03:23:25 on dist-test-slave-gw2v"
server_key: "f5a141fb3f18214261bae844457f28d1"
server_key_iv: "c42ea6fbc69f075ddf0377f112530ce2"
server_key_version: "encryptionkey@0"
I20260927 03:23:25.331935 7976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:25.352224 7976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:25.354233 7976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:25.354358 7976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:25.354918 7976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:25.354964 7976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.354990 7976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:25.355001 7976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.361886 7976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44905
I20260927 03:23:25.361943 8106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44905 every 8 connection(s)
Sep 27 03:23:25 dist-test-slave-gw2v krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479405, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:23:25.362339 7976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:23:25.365200 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.355493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35577 (local address 127.0.98.62:39957)
0927 03:23:25.355542 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.355544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.355724 (+ 180us) server_negotiation.cc:407] Connection header received
0927 03:23:25.356432 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.356433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.356447 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.356500 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:25.357405 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.358252 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.359789 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.360006 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.360060 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.362727 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:25.362729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:25.362729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:25.362740 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:25.363919 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.364323 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.364324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.364324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.364455 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.364650 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.364651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.364652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.364761 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:25.364808 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.365049 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.365135 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:23:25.365314 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.355393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39957 (local address 127.0.98.1:35577)
0927 03:23:25.355579 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.355629 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.356288 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.356608 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.356613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.356995 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:25.357301 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.357308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.358350 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.358351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.359705 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.359707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.360065 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.360066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.360209 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.360832 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:25.360845 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:25.362608 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:25.364013 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:25.364018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:25.364021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:25.364235 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:25.364515 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:25.364515 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:25.364516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:25.364605 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:25.364804 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:25.364806 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:25.364920 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.365022 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.365142 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260927 03:23:25.365864 8107 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39957
I20260927 03:23:25.365949 8107 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:25.366127 8107 heartbeater.cc:511] Master 127.0.98.62:39957 requested a full tablet report, sending...
I20260927 03:23:25.366711 7903 ts_manager.cc:194] Registered new tserver with Master: f4aef3b67d7a4d048bb7c4e1a0655d09 (127.0.98.1:44905)
I20260927 03:23:25.366954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 7976
I20260927 03:23:25.367116 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:23:25.367357 392 external_mini_cluster.cc:1428] Setting key df8b6bd115320b684b90c26e6f5502fb
I20260927 03:23:25.367808 7903 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35577
I20260927 03:23:25.374612 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.367947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39957 (local address 127.0.98.1:52059)
0927 03:23:25.369149 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.369159 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.369218 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.369326 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.369327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.369507 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:25.369556 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.369558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.370201 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.370201 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.370964 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.370967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.371290 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.371291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.371361 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.371831 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:25.371836 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:25.372179 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:25.373686 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:25.373687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:25.373688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:25.373845 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:25.374015 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:25.374016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:25.374016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:25.374072 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:25.374287 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:25.374288 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:25.374334 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.374392 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.374497 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"thread_start_us":77,"threads_started":1}
I20260927 03:23:25.374604 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.368034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52059 (local address 127.0.98.62:39957)
0927 03:23:25.368070 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.368072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.369226 (+ 1154us) server_negotiation.cc:407] Connection header received
0927 03:23:25.369268 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.369269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.369279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.369323 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:25.369624 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.370110 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.371036 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.371222 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.371298 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.372253 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:25.372254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:25.372255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:25.372261 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:25.373608 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.373904 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.373904 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.373905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.373960 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.374119 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.374120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.374120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.374234 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:25.374288 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.374419 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.374522 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 03:23:25.394397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39957
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:25.476027 8116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:25.476245 8116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:25.476270 8116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:25.477474 8116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:25.477523 8116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:25.477543 8116 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:23:25.477561 8116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:23:25.479121 8116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:25.479348 8116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:25.479570 8116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:25.484828 8122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:25.484828 8124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:25.484900 8121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:25.485028 8116 server_base.cc:1034] running on GCE node
I20260927 03:23:25.485325 8116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:25.485622 8116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:25.486799 8116 hybrid_clock.cc:648] HybridClock initialized: now 1790479405486786 us; error 32 us; skew 500 ppm
Sep 27 03:23:25 dist-test-slave-gw2v krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479405, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:25.489523 8116 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 03:23:25.489961 8116 webserver.cc:460] Webserver started at http://127.0.98.2:45987/ using document root <none> and password file <none>
I20260927 03:23:25.490139 8116 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:25.490177 8116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:25.490271 8116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:25.491405 8116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "f5283d7913c4447085236c190fd059ef"
format_stamp: "Formatted at 2026-09-27 03:23:25 on dist-test-slave-gw2v"
server_key: "356c8de2b8480e95979d69045a5d5008"
server_key_iv: "b4058c2f3b5bbfc4508ccc3d20727557"
server_key_version: "encryptionkey@0"
I20260927 03:23:25.491730 8116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f5283d7913c4447085236c190fd059ef"
format_stamp: "Formatted at 2026-09-27 03:23:25 on dist-test-slave-gw2v"
server_key: "356c8de2b8480e95979d69045a5d5008"
server_key_iv: "b4058c2f3b5bbfc4508ccc3d20727557"
server_key_version: "encryptionkey@0"
I20260927 03:23:25.493039 8116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 03:23:25.493916 8131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.494108 8116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:25.494194 8116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "f5283d7913c4447085236c190fd059ef"
format_stamp: "Formatted at 2026-09-27 03:23:25 on dist-test-slave-gw2v"
server_key: "356c8de2b8480e95979d69045a5d5008"
server_key_iv: "b4058c2f3b5bbfc4508ccc3d20727557"
server_key_version: "encryptionkey@0"
I20260927 03:23:25.494254 8116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:25.506466 8116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:25.508744 8116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:25.508874 8116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:25.509418 8116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:25.509473 8116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.509495 8116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:25.509506 8116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.516330 8116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33143
I20260927 03:23:25.516403 8246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33143 every 8 connection(s)
Sep 27 03:23:25 dist-test-slave-gw2v krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479405, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:23:25.516716 8116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:23:25.519513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 8116
I20260927 03:23:25.519601 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:23:25.519775 392 external_mini_cluster.cc:1428] Setting key 1f46a7c8926224bfbdb7432e70777a22
I20260927 03:23:25.519968 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.509905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35725 (local address 127.0.98.62:39957)
0927 03:23:25.509945 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.509947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.510181 (+ 234us) server_negotiation.cc:407] Connection header received
0927 03:23:25.510723 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.510724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.510737 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.510804 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:25.511742 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.512514 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.514119 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.514415 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.514478 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.517168 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:25.517170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:25.517170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:25.517179 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:25.518604 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.519038 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.519039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.519040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.519091 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.519310 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.519311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.519311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.519426 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:25.519479 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.519790 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.519868 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:23:25.520135 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.509919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39957 (local address 127.0.98.2:35725)
0927 03:23:25.510067 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.510102 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.510621 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.510942 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.510947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.511319 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:25.511630 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.511637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.512634 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.512636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.514027 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.514030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.514500 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.514502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.514654 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.515345 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:25.515356 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:25.517068 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:25.518682 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:25.518688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:25.518691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:25.518965 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:25.519142 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:25.519142 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:25.519143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:25.519249 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:25.519479 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:25.519485 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:25.519650 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.519745 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.519914 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260927 03:23:25.520704 8247 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39957
I20260927 03:23:25.520793 8247 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:25.520982 8247 heartbeater.cc:511] Master 127.0.98.62:39957 requested a full tablet report, sending...
I20260927 03:23:25.521343 7904 ts_manager.cc:194] Registered new tserver with Master: f5283d7913c4447085236c190fd059ef (127.0.98.2:33143)
I20260927 03:23:25.521793 7904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35725
I20260927 03:23:25.527211 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.521748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42569 (local address 127.0.98.62:39957)
0927 03:23:25.521809 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.521811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.522098 (+ 287us) server_negotiation.cc:407] Connection header received
0927 03:23:25.522152 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.522153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.522161 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.522214 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:25.522529 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.523058 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.523727 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.523931 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.523984 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.524838 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:25.524840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:25.524840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:25.524847 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:25.526100 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.526398 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.526399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.526399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.526456 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.526614 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.526615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.526615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.526756 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:25.526872 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.527030 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.527105 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:23:25.527220 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.521791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39957 (local address 127.0.98.2:42569)
0927 03:23:25.522034 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.522046 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.522099 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.522230 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.522232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.522414 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:25.522463 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.522464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.523146 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.523147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.523659 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.523661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.523987 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.523988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.524051 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.524466 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:25.524471 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:25.524759 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:25.526170 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:25.526171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:25.526171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:25.526341 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:25.526509 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:25.526510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:25.526510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:25.526567 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:25.526900 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:25.526901 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:25.526943 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.526998 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.527106 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":178,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 03:23:25.547448 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39957
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:25.630262 8256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:25.630476 8256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:25.630503 8256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:25.631712 8256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:25.631757 8256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:25.631778 8256 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:23:25.631794 8256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:23:25.633306 8256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:25.633543 8256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:25.633749 8256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:25.639063 8262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:25.639074 8261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:25.639123 8256 server_base.cc:1034] running on GCE node
W20260927 03:23:25.639061 8264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:25.639446 8256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:25.639673 8256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:25.640815 8256 hybrid_clock.cc:648] HybridClock initialized: now 1790479405640805 us; error 31 us; skew 500 ppm
Sep 27 03:23:25 dist-test-slave-gw2v krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479405, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:25.643441 8256 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 03:23:25.643958 8256 webserver.cc:460] Webserver started at http://127.0.98.3:44015/ using document root <none> and password file <none>
I20260927 03:23:25.644162 8256 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:25.644210 8256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:25.644352 8256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:25.645406 8256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "aa6a9f51988347d49434496e6e08d2d4"
format_stamp: "Formatted at 2026-09-27 03:23:25 on dist-test-slave-gw2v"
server_key: "0852425271709dbb679aed0b5d983f3c"
server_key_iv: "a52997b2dd042c9655415e9ec313d5a5"
server_key_version: "encryptionkey@0"
I20260927 03:23:25.645740 8256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aa6a9f51988347d49434496e6e08d2d4"
format_stamp: "Formatted at 2026-09-27 03:23:25 on dist-test-slave-gw2v"
server_key: "0852425271709dbb679aed0b5d983f3c"
server_key_iv: "a52997b2dd042c9655415e9ec313d5a5"
server_key_version: "encryptionkey@0"
I20260927 03:23:25.647089 8256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:23:25.647802 8271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.647945 8256 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:25.648030 8256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "aa6a9f51988347d49434496e6e08d2d4"
format_stamp: "Formatted at 2026-09-27 03:23:25 on dist-test-slave-gw2v"
server_key: "0852425271709dbb679aed0b5d983f3c"
server_key_iv: "a52997b2dd042c9655415e9ec313d5a5"
server_key_version: "encryptionkey@0"
I20260927 03:23:25.648119 8256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:25.666656 8256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:25.669207 8256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:25.669356 8256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:25.669958 8256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:25.670001 8256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.670030 8256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:25.670045 8256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.676129 8256 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33389
I20260927 03:23:25.676261 8386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33389 every 8 connection(s)
I20260927 03:23:25.676503 8256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:23:25 dist-test-slave-gw2v krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479405, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:23:25.679715 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.670409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40533 (local address 127.0.98.62:39957)
0927 03:23:25.670453 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.670455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.670676 (+ 221us) server_negotiation.cc:407] Connection header received
0927 03:23:25.671289 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.671291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.671305 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.671379 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:25.672186 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.672978 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.674457 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.674752 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.674848 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.677336 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:25.677338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:25.677338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:25.677346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:25.678428 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.678798 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.678799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.678800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.678884 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.679104 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.679105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.679105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.679223 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:25.679278 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.679543 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.679626 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:23:25.679885 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.670415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39957 (local address 127.0.98.3:40533)
0927 03:23:25.670559 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.670595 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.671168 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.671464 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.671468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.671801 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:25.672087 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.672093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.673070 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.673071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.674363 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.674367 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.674865 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.674866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.675020 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.675604 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:25.675623 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:25.677245 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:25.678508 (+ 1263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:25.678514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:25.678517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:25.678731 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:25.678943 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:25.678943 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:25.678944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:25.679050 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:25.679275 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:25.679281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:25.679423 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.679512 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.679668 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260927 03:23:25.680347 8387 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39957
I20260927 03:23:25.680488 8387 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:25.680660 8387 heartbeater.cc:511] Master 127.0.98.62:39957 requested a full tablet report, sending...
I20260927 03:23:25.681067 7903 ts_manager.cc:194] Registered new tserver with Master: aa6a9f51988347d49434496e6e08d2d4 (127.0.98.3:33389)
I20260927 03:23:25.681504 7903 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40533
I20260927 03:23:25.682871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 8256
I20260927 03:23:25.683007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:23:25.683164 392 external_mini_cluster.cc:1428] Setting key 227868785b5ab7914db0c72177b21516
I20260927 03:23:25.683909 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:23:25.687104 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.681411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47027 (local address 127.0.98.62:39957)
0927 03:23:25.681453 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.681454 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.681515 (+ 61us) server_negotiation.cc:407] Connection header received
0927 03:23:25.681597 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.681599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.681608 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.681662 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:25.682038 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.682588 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.683485 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.683744 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.683806 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.684725 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:25.684726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:25.684726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:25.684734 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:25.686040 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.686418 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.686419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.686419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.686466 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.686669 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.686670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.686670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.686768 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:25.686813 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.686978 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.687040 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:23:25.687165 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.681330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39957 (local address 127.0.98.3:47027)
0927 03:23:25.681459 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.681469 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.681529 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.681667 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.681668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.681853 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:25.681942 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.681944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.682694 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.682695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.683435 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.683437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.683814 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.683815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.683879 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.684294 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:25.684298 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:25.684641 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:25.686122 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:25.686124 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:25.686124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:25.686338 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:25.686517 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:25.686518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:25.686518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:25.686593 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:25.686841 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:25.686842 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:25.686889 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.686961 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.687064 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
Sep 27 03:23:25 dist-test-slave-gw2v krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479405, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:23:25 dist-test-slave-gw2v krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:25 dist-test-slave-gw2v krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:23:25.706420 7972 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:25.700432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54224 (local address 127.0.98.62:39957)
0927 03:23:25.700485 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.700487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.700510 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:23:25.700574 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.700575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.700584 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.700637 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:25.701387 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.701903 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.702674 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.702913 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.702981 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.706333 (+ 3352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54224: BlockingRecv error: recv got EOF from 127.0.0.1:54224 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 03:23:25 dist-test-slave-gw2v krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479405, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:23:25.714597 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.707303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.0.98.62:39957)
0927 03:23:25.707422 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.707424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.707458 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:23:25.707482 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.707482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.707491 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.707533 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:25.707945 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.708480 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.709195 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.709440 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.709498 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.711972 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:25.711974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:25.711974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:25.711981 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:25.713310 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.713674 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.713676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.713676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.713739 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.713983 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.713984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.713984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.714108 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:25.714197 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.714417 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.714494 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 03:23:25.716389 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:23:25.718245 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:23:25.724913 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.720887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.0.98.1:44905)
0927 03:23:25.721050 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.721055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.721185 (+ 130us) server_negotiation.cc:407] Connection header received
0927 03:23:25.721469 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.721472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.721487 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.721552 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:25.721677 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.722517 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.723932 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.724480 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.724515 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.724571 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.724773 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.724814 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260927 03:23:25.724913 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.720766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44905 (local address 127.0.0.1:55806)
0927 03:23:25.721073 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.721122 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.721227 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.721560 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.721561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.721563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:25.721598 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.721599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.722605 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.722606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.723837 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.723839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.724617 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.724617 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.724674 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.724684 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.724739 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.724803 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":71,"threads_started":1}
I20260927 03:23:25.725090 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.721198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.0.98.2:33143)
0927 03:23:25.721377 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.721381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.721389 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:25.721427 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.721429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.721443 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.721519 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:25.721710 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.722844 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.723837 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.724542 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.724609 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.724679 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.724953 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.724994 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
I20260927 03:23:25.725349 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.720934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.0.98.3:33389)
0927 03:23:25.721132 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.721136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.721223 (+ 87us) server_negotiation.cc:407] Connection header received
0927 03:23:25.721373 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.721376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.721394 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.721471 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:25.721648 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.722853 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.724075 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.724770 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.725133 (+ 363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.725219 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.725243 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.725279 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
I20260927 03:23:25.725060 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.720766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33389 (local address 127.0.0.1:50010)
0927 03:23:25.721154 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.721164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.721252 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.721496 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.721499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.721501 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:25.721572 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.721574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.722921 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.722922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.724015 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.724017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.724868 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.724869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.724935 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.724938 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.724967 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.724997 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":8,"thread_start_us":102,"threads_started":1}
I20260927 03:23:25.725334 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.720991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33143 (local address 127.0.0.1:40800)
0927 03:23:25.721183 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.721191 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.721230 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.721585 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.721586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.721588 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:25.721635 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.721637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.722951 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.722952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.723769 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.723771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.724712 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.724713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.724771 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.724776 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.724820 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.724862 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260927 03:23:25.726094 8321 tablet_service.cc:1467] Processing CreateTablet for tablet 85b0376e64974b539061ed7b728f42f9 (DEFAULT_TABLE table=test-table [id=b187c0d2808144d6966b84f66777d6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:25.726105 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 85b0376e64974b539061ed7b728f42f9 (DEFAULT_TABLE table=test-table [id=b187c0d2808144d6966b84f66777d6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:25.726255 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 85b0376e64974b539061ed7b728f42f9 (DEFAULT_TABLE table=test-table [id=b187c0d2808144d6966b84f66777d6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:25.726498 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85b0376e64974b539061ed7b728f42f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:25.726507 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85b0376e64974b539061ed7b728f42f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:25.726540 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85b0376e64974b539061ed7b728f42f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:25.729890 8415 tablet_bootstrap.cc:492] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09: Bootstrap starting.
I20260927 03:23:25.730233 8416 tablet_bootstrap.cc:492] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4: Bootstrap starting.
I20260927 03:23:25.730437 8417 tablet_bootstrap.cc:492] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef: Bootstrap starting.
I20260927 03:23:25.730646 8415 tablet_bootstrap.cc:654] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:25.730863 8416 tablet_bootstrap.cc:654] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:25.731004 8415 log.cc:824] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:25.731034 8417 tablet_bootstrap.cc:654] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:25.731142 8416 log.cc:824] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:25.731323 8417 log.cc:824] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:25.731832 8416 tablet_bootstrap.cc:492] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4: No bootstrap required, opened a new log
I20260927 03:23:25.731890 8416 ts_tablet_manager.cc:1397] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:23:25.732005 8417 tablet_bootstrap.cc:492] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef: No bootstrap required, opened a new log
I20260927 03:23:25.732059 8417 ts_tablet_manager.cc:1397] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:23:25.733407 8416 raft_consensus.cc:348] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } }
I20260927 03:23:25.733546 8416 raft_consensus.cc:374] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:25.733577 8416 raft_consensus.cc:729] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6a9f51988347d49434496e6e08d2d4, State: Initialized, Role: FOLLOWER
I20260927 03:23:25.733520 8417 raft_consensus.cc:348] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } }
I20260927 03:23:25.733659 8417 raft_consensus.cc:374] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:25.733696 8417 raft_consensus.cc:729] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5283d7913c4447085236c190fd059ef, State: Initialized, Role: FOLLOWER
I20260927 03:23:25.733700 8416 consensus_queue.cc:260] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } }
I20260927 03:23:25.733806 8417 consensus_queue.cc:260] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } }
I20260927 03:23:25.733954 8416 ts_tablet_manager.cc:1428] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:23:25.733979 8387 heartbeater.cc:503] Master 127.0.98.62:39957 was elected leader, sending a full tablet report...
I20260927 03:23:25.734064 8417 ts_tablet_manager.cc:1428] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:23:25.734170 8247 heartbeater.cc:503] Master 127.0.98.62:39957 was elected leader, sending a full tablet report...
I20260927 03:23:25.734500 8415 tablet_bootstrap.cc:492] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09: No bootstrap required, opened a new log
I20260927 03:23:25.734578 8415 ts_tablet_manager.cc:1397] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 03:23:25.736339 8415 raft_consensus.cc:348] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } }
I20260927 03:23:25.736475 8415 raft_consensus.cc:374] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:25.736495 8415 raft_consensus.cc:729] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4aef3b67d7a4d048bb7c4e1a0655d09, State: Initialized, Role: FOLLOWER
I20260927 03:23:25.736594 8415 consensus_queue.cc:260] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } }
I20260927 03:23:25.736881 8415 ts_tablet_manager.cc:1428] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:23:25.736919 8107 heartbeater.cc:503] Master 127.0.98.62:39957 was elected leader, sending a full tablet report...
I20260927 03:23:25.743741 8421 raft_consensus.cc:482] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:25.743848 8421 raft_consensus.cc:504] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } }
I20260927 03:23:25.744210 8421 leader_election.cc:288] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4aef3b67d7a4d048bb7c4e1a0655d09 (127.0.98.1:44905), f5283d7913c4447085236c190fd059ef (127.0.98.2:33143)
I20260927 03:23:25.747115 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.744458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46671 (local address 127.0.98.1:44905)
0927 03:23:25.744511 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.744513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.744523 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:25.744580 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.744580 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.744591 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.744649 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:25.744923 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.745439 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.746398 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.746758 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.746839 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.746851 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.747013 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.747048 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:23:25.747117 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.744605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40613 (local address 127.0.98.2:33143)
0927 03:23:25.744652 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.744654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.744763 (+ 109us) server_negotiation.cc:407] Connection header received
0927 03:23:25.744808 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.744809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.744820 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.744876 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:25.745013 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.745454 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.746410 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.746770 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.746857 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.746866 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.747010 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.747050 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:23:25.747133 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.744414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44905 (local address 127.0.98.3:46671)
0927 03:23:25.744472 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.744484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.744527 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.744696 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.744696 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.744698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:25.744764 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.744766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.745515 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.745516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.746314 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.746316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.746856 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.746857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.746946 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.746950 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.747003 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.747046 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":27}
I20260927 03:23:25.747144 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.744530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33143 (local address 127.0.98.3:40613)
0927 03:23:25.744695 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.744705 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.744756 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.744881 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.744881 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.744883 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:25.744950 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.744951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.745524 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.745525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.746314 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.746316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.746862 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.746863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.746947 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.746950 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.746996 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.747038 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":21,"thread_start_us":115,"threads_started":1}
I20260927 03:23:25.747499 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b0376e64974b539061ed7b728f42f9" candidate_uuid: "aa6a9f51988347d49434496e6e08d2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" is_pre_election: true
I20260927 03:23:25.747663 8061 raft_consensus.cc:2454] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa6a9f51988347d49434496e6e08d2d4 in term 0.
I20260927 03:23:25.747627 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b0376e64974b539061ed7b728f42f9" candidate_uuid: "aa6a9f51988347d49434496e6e08d2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5283d7913c4447085236c190fd059ef" is_pre_election: true
I20260927 03:23:25.747766 8201 raft_consensus.cc:2454] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa6a9f51988347d49434496e6e08d2d4 in term 0.
I20260927 03:23:25.747869 8272 leader_election.cc:302] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa6a9f51988347d49434496e6e08d2d4, f4aef3b67d7a4d048bb7c4e1a0655d09; no voters:
I20260927 03:23:25.748004 8421 raft_consensus.cc:2781] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:23:25.748066 8421 raft_consensus.cc:482] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:25.748081 8421 raft_consensus.cc:3037] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:25.749101 8421 raft_consensus.cc:504] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } }
I20260927 03:23:25.749233 8421 leader_election.cc:288] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [CANDIDATE]: Term 1 election: Requested vote from peers f4aef3b67d7a4d048bb7c4e1a0655d09 (127.0.98.1:44905), f5283d7913c4447085236c190fd059ef (127.0.98.2:33143)
I20260927 03:23:25.749394 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b0376e64974b539061ed7b728f42f9" candidate_uuid: "aa6a9f51988347d49434496e6e08d2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5283d7913c4447085236c190fd059ef"
I20260927 03:23:25.749433 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b0376e64974b539061ed7b728f42f9" candidate_uuid: "aa6a9f51988347d49434496e6e08d2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09"
I20260927 03:23:25.749473 8201 raft_consensus.cc:3037] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:25.749537 8061 raft_consensus.cc:3037] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:25.750133 8201 raft_consensus.cc:2454] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa6a9f51988347d49434496e6e08d2d4 in term 1.
I20260927 03:23:25.750281 8272 leader_election.cc:302] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa6a9f51988347d49434496e6e08d2d4, f5283d7913c4447085236c190fd059ef; no voters:
I20260927 03:23:25.750317 8061 raft_consensus.cc:2454] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa6a9f51988347d49434496e6e08d2d4 in term 1.
I20260927 03:23:25.750360 8421 raft_consensus.cc:2781] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:25.750514 8421 raft_consensus.cc:686] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [term 1 LEADER]: Becoming Leader. State: Replica: aa6a9f51988347d49434496e6e08d2d4, State: Running, Role: LEADER
I20260927 03:23:25.750609 8421 consensus_queue.cc:237] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } }
I20260927 03:23:25.751447 7903 catalog_manager.cc:5495] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa6a9f51988347d49434496e6e08d2d4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "aa6a9f51988347d49434496e6e08d2d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } health_report { overall_health: UNKNOWN } } }
I20260927 03:23:25.760720 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.758335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.0.98.3:33389)
0927 03:23:25.758425 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.758427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.758434 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:23:25.758462 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.758463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.758479 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.758536 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:25.758672 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.759178 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.760007 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.760206 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.760275 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.760448 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:25.760558 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:25.760613 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.760630 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.760646 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:23:25.762585 8061 raft_consensus.cc:1261] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Refusing update from remote peer aa6a9f51988347d49434496e6e08d2d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:25.762595 8201 raft_consensus.cc:1261] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Refusing update from remote peer aa6a9f51988347d49434496e6e08d2d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:25.762950 8425 consensus_queue.cc:1035] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:25.763084 8421 consensus_queue.cc:1035] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:25.765969 8429 mvcc.cc:205] Tried to move back new op lower bound from 7333803646001008640 to 7333803645955215360. Current Snapshot: MvccSnapshot[applied={T|T < 7333803646001008640}]
I20260927 03:23:25.766193 8430 mvcc.cc:205] Tried to move back new op lower bound from 7333803646001008640 to 7333803645955215360. Current Snapshot: MvccSnapshot[applied={T|T < 7333803646001008640 or (T in {7333803646001008640})}]
I20260927 03:23:25.767578 7903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:25.767679 7903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:25.768383 7903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a: Sending DeleteTablet for 3 replicas of tablet 85b0376e64974b539061ed7b728f42f9
I20260927 03:23:25.768676 8321 tablet_service.cc:1514] Processing DeleteTablet for tablet 85b0376e64974b539061ed7b728f42f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50010
I20260927 03:23:25.768714 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 85b0376e64974b539061ed7b728f42f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40800
I20260927 03:23:25.768715 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 85b0376e64974b539061ed7b728f42f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55806
I20260927 03:23:25.769259 8439 tablet_replica.cc:330] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef: stopping tablet replica
I20260927 03:23:25.769217 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:23:25.769356 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:23:25.769362 8439 raft_consensus.cc:2229] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:25.769469 8439 raft_consensus.cc:2258] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:25.769901 8439 ts_tablet_manager.cc:1905] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:25.770535 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 71e286fc43e9485d8ba9045331c562c3 (DEFAULT_TABLE table=test-table [id=0f0f97d9565a43478252ec96b12901b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:25.770535 8321 tablet_service.cc:1467] Processing CreateTablet for tablet 71e286fc43e9485d8ba9045331c562c3 (DEFAULT_TABLE table=test-table [id=0f0f97d9565a43478252ec96b12901b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:25.770642 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e286fc43e9485d8ba9045331c562c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:25.770642 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e286fc43e9485d8ba9045331c562c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:25.771657 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 71e286fc43e9485d8ba9045331c562c3 (DEFAULT_TABLE table=test-table [id=0f0f97d9565a43478252ec96b12901b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:25.771763 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e286fc43e9485d8ba9045331c562c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:25.772045 8417 tablet_bootstrap.cc:492] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef: Bootstrap starting.
I20260927 03:23:25.772030 8416 tablet_bootstrap.cc:492] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4: Bootstrap starting.
I20260927 03:23:25.772387 8417 tablet_bootstrap.cc:654] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:25.772432 8416 tablet_bootstrap.cc:654] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:25.772914 8417 tablet_bootstrap.cc:492] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef: No bootstrap required, opened a new log
I20260927 03:23:25.772957 8417 ts_tablet_manager.cc:1397] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:25.773092 8417 raft_consensus.cc:348] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.773167 8417 raft_consensus.cc:374] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:25.773186 8417 raft_consensus.cc:729] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5283d7913c4447085236c190fd059ef, State: Initialized, Role: FOLLOWER
I20260927 03:23:25.773226 8417 consensus_queue.cc:260] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.773319 8417 ts_tablet_manager.cc:1428] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.773331 8415 tablet_bootstrap.cc:492] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09: Bootstrap starting.
I20260927 03:23:25.773706 8415 tablet_bootstrap.cc:654] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:25.774276 8415 tablet_bootstrap.cc:492] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09: No bootstrap required, opened a new log
I20260927 03:23:25.774330 8415 ts_tablet_manager.cc:1397] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:23:25.774474 8415 raft_consensus.cc:348] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.774544 8415 raft_consensus.cc:374] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:25.774575 8415 raft_consensus.cc:729] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4aef3b67d7a4d048bb7c4e1a0655d09, State: Initialized, Role: FOLLOWER
I20260927 03:23:25.774632 8415 consensus_queue.cc:260] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.774718 8415 ts_tablet_manager.cc:1428] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.774901 8439 ts_tablet_manager.cc:1918] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:25.774991 8439 log.cc:1196] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/85b0376e64974b539061ed7b728f42f9
I20260927 03:23:25.775238 8439 ts_tablet_manager.cc:1939] T 85b0376e64974b539061ed7b728f42f9 P f5283d7913c4447085236c190fd059ef: Deleting consensus metadata
I20260927 03:23:25.775619 8438 tablet_replica.cc:330] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09: stopping tablet replica
I20260927 03:23:25.775626 7888 catalog_manager.cc:4841] TS f5283d7913c4447085236c190fd059ef (127.0.98.2:33143): tablet 85b0376e64974b539061ed7b728f42f9 (table test-table [id=b187c0d2808144d6966b84f66777d6dd]) successfully deleted
I20260927 03:23:25.775736 8438 raft_consensus.cc:2229] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:25.775844 8438 raft_consensus.cc:2258] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:25.776240 8438 ts_tablet_manager.cc:1905] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:25.777769 8438 ts_tablet_manager.cc:1918] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:25.777838 8438 log.cc:1196] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/wal/wals/85b0376e64974b539061ed7b728f42f9
I20260927 03:23:25.778079 8438 ts_tablet_manager.cc:1939] T 85b0376e64974b539061ed7b728f42f9 P f4aef3b67d7a4d048bb7c4e1a0655d09: Deleting consensus metadata
I20260927 03:23:25.778534 7888 catalog_manager.cc:4841] TS f4aef3b67d7a4d048bb7c4e1a0655d09 (127.0.98.1:44905): tablet 85b0376e64974b539061ed7b728f42f9 (table test-table [id=b187c0d2808144d6966b84f66777d6dd]) successfully deleted
I20260927 03:23:25.778944 8437 tablet_replica.cc:330] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4: stopping tablet replica
I20260927 03:23:25.779052 8437 raft_consensus.cc:2229] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:23:25.779160 8437 raft_consensus.cc:2258] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:25.779199 8416 tablet_bootstrap.cc:492] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4: No bootstrap required, opened a new log
I20260927 03:23:25.779255 8416 ts_tablet_manager.cc:1397] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260927 03:23:25.779412 8416 raft_consensus.cc:348] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.779488 8416 raft_consensus.cc:374] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:25.779507 8416 raft_consensus.cc:729] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6a9f51988347d49434496e6e08d2d4, State: Initialized, Role: FOLLOWER
I20260927 03:23:25.779554 8416 consensus_queue.cc:260] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.779636 8416 ts_tablet_manager.cc:1428] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.779798 8437 ts_tablet_manager.cc:1905] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:25.781419 8437 ts_tablet_manager.cc:1918] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:25.781490 8437 log.cc:1196] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/85b0376e64974b539061ed7b728f42f9
I20260927 03:23:25.781708 8437 ts_tablet_manager.cc:1939] T 85b0376e64974b539061ed7b728f42f9 P aa6a9f51988347d49434496e6e08d2d4: Deleting consensus metadata
I20260927 03:23:25.782078 7889 catalog_manager.cc:4841] TS aa6a9f51988347d49434496e6e08d2d4 (127.0.98.3:33389): tablet 85b0376e64974b539061ed7b728f42f9 (table test-table [id=b187c0d2808144d6966b84f66777d6dd]) successfully deleted
I20260927 03:23:25.811904 8423 raft_consensus.cc:482] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:25.812023 8423 raft_consensus.cc:504] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.812413 8423 leader_election.cc:288] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5283d7913c4447085236c190fd059ef (127.0.98.2:33143), aa6a9f51988347d49434496e6e08d2d4 (127.0.98.3:33389)
I20260927 03:23:25.815891 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.812767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58701 (local address 127.0.98.3:33389)
0927 03:23:25.812849 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.812851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.813020 (+ 169us) server_negotiation.cc:407] Connection header received
0927 03:23:25.813074 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.813075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.813084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.813124 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:25.813323 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.814086 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.815194 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.815563 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.815632 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.815642 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.815781 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.815817 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:23:25.815923 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.812673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33389 (local address 127.0.98.1:58701)
0927 03:23:25.812966 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.812974 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.813024 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.813166 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.813167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.813169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:25.813252 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.813253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.814201 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.814203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.815106 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.815109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.815640 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.815641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.815712 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.815717 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.815768 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.815809 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":187,"thread_start_us":65,"threads_started":1}
I20260927 03:23:25.816228 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e286fc43e9485d8ba9045331c562c3" candidate_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6a9f51988347d49434496e6e08d2d4" is_pre_election: true
I20260927 03:23:25.816391 8341 raft_consensus.cc:2454] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4aef3b67d7a4d048bb7c4e1a0655d09 in term 0.
I20260927 03:23:25.816440 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.812673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33143 (local address 127.0.98.1:39491)
0927 03:23:25.812724 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.812735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.812792 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.812977 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.812978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.812979 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:25.813056 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.813058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.814167 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.814168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.815442 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.815445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.816151 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.816152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.816226 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.816230 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.816303 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.816373 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":38}
I20260927 03:23:25.816619 7993 leader_election.cc:302] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa6a9f51988347d49434496e6e08d2d4, f4aef3b67d7a4d048bb7c4e1a0655d09; no voters:
I20260927 03:23:25.816725 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.812804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39491 (local address 127.0.98.2:33143)
0927 03:23:25.812849 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.812851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.812861 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:25.812890 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.812891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.812899 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.812966 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:25.813193 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.814077 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.815538 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.816014 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.816082 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.816092 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.816619 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.816665 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:23:25.816757 8423 raft_consensus.cc:2781] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:23:25.816785 8423 raft_consensus.cc:482] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:25.816803 8423 raft_consensus.cc:3037] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:25.816911 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e286fc43e9485d8ba9045331c562c3" candidate_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5283d7913c4447085236c190fd059ef" is_pre_election: true
I20260927 03:23:25.817003 8201 raft_consensus.cc:2454] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4aef3b67d7a4d048bb7c4e1a0655d09 in term 0.
I20260927 03:23:25.817565 8423 raft_consensus.cc:504] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.817695 8423 leader_election.cc:288] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [CANDIDATE]: Term 1 election: Requested vote from peers f5283d7913c4447085236c190fd059ef (127.0.98.2:33143), aa6a9f51988347d49434496e6e08d2d4 (127.0.98.3:33389)
I20260927 03:23:25.817843 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e286fc43e9485d8ba9045331c562c3" candidate_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6a9f51988347d49434496e6e08d2d4"
I20260927 03:23:25.817840 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e286fc43e9485d8ba9045331c562c3" candidate_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5283d7913c4447085236c190fd059ef"
I20260927 03:23:25.817920 8341 raft_consensus.cc:3037] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:25.817925 8201 raft_consensus.cc:3037] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:25.818593 8201 raft_consensus.cc:2454] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4aef3b67d7a4d048bb7c4e1a0655d09 in term 1.
I20260927 03:23:25.818737 8341 raft_consensus.cc:2454] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4aef3b67d7a4d048bb7c4e1a0655d09 in term 1.
I20260927 03:23:25.818761 7992 leader_election.cc:302] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f4aef3b67d7a4d048bb7c4e1a0655d09, f5283d7913c4447085236c190fd059ef; no voters:
I20260927 03:23:25.818862 8423 raft_consensus.cc:2781] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:25.818982 8423 raft_consensus.cc:686] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 LEADER]: Becoming Leader. State: Replica: f4aef3b67d7a4d048bb7c4e1a0655d09, State: Running, Role: LEADER
I20260927 03:23:25.819051 8423 consensus_queue.cc:237] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.819747 7904 catalog_manager.cc:5495] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4aef3b67d7a4d048bb7c4e1a0655d09 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } health_report { overall_health: UNKNOWN } } }
W20260927 03:23:25.863137 8108 tablet.cc:2369] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 03:23:25 dist-test-slave-gw2v krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479405, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:23:25.883208 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.875532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39957 (local address 127.0.0.1:54238)
0927 03:23:25.875699 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.875708 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.875766 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.875929 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.875929 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.876215 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:25.876261 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.876263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.877194 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.877196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.877788 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.877790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.878422 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.878423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.878476 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.879082 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:25.879087 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:25.880468 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:25.882167 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:25.882175 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:25.882177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:25.882448 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:25.882633 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:25.882633 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:25.882634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:25.882693 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:25.882942 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:25.882948 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:25.882993 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.883042 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.883133 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
I20260927 03:23:25.883307 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.875633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54238 (local address 127.0.98.62:39957)
0927 03:23:25.875695 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.875697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.875765 (+ 68us) server_negotiation.cc:407] Connection header received
0927 03:23:25.875833 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.875834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.875845 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.875900 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:25.876342 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.877099 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.877883 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.878189 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.878266 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.880547 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:25.880549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:25.880549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:25.880557 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:25.882086 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.882517 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.882518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.882519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.882576 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:25.882739 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:25.882740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:25.882740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:25.882880 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:25.882931 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.883076 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.883191 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":35}
I20260927 03:23:25.885324 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54238:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 03:23:25.885540 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:23:25.886859 8181 tablet_service.cc:1467] Processing CreateTablet for tablet e638def81f754691a7d9f92619d70016 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=77f1dc0b1de743ddae0b7009c014f64c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:23:25.886906 8321 tablet_service.cc:1467] Processing CreateTablet for tablet e638def81f754691a7d9f92619d70016 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=77f1dc0b1de743ddae0b7009c014f64c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:23:25.886952 8041 tablet_service.cc:1467] Processing CreateTablet for tablet e638def81f754691a7d9f92619d70016 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=77f1dc0b1de743ddae0b7009c014f64c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:23:25.887020 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e638def81f754691a7d9f92619d70016. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:25.887019 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e638def81f754691a7d9f92619d70016. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:25.887087 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e638def81f754691a7d9f92619d70016. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:25.888159 8415 tablet_bootstrap.cc:492] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09: Bootstrap starting.
I20260927 03:23:25.888405 8416 tablet_bootstrap.cc:492] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4: Bootstrap starting.
I20260927 03:23:25.888414 8417 tablet_bootstrap.cc:492] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef: Bootstrap starting.
I20260927 03:23:25.888509 8415 tablet_bootstrap.cc:654] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:25.888672 8416 tablet_bootstrap.cc:654] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:25.888762 8417 tablet_bootstrap.cc:654] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:25.889132 8416 tablet_bootstrap.cc:492] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4: No bootstrap required, opened a new log
I20260927 03:23:25.889142 8415 tablet_bootstrap.cc:492] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09: No bootstrap required, opened a new log
I20260927 03:23:25.889171 8416 ts_tablet_manager.cc:1397] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:23:25.889178 8415 ts_tablet_manager.cc:1397] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:23:25.889276 8416 raft_consensus.cc:348] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.889300 8415 raft_consensus.cc:348] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.889360 8415 raft_consensus.cc:374] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:25.889360 8416 raft_consensus.cc:374] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:25.889381 8416 raft_consensus.cc:729] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6a9f51988347d49434496e6e08d2d4, State: Initialized, Role: FOLLOWER
I20260927 03:23:25.889381 8415 raft_consensus.cc:729] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4aef3b67d7a4d048bb7c4e1a0655d09, State: Initialized, Role: FOLLOWER
I20260927 03:23:25.889420 8415 consensus_queue.cc:260] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.889420 8416 consensus_queue.cc:260] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.889444 8417 tablet_bootstrap.cc:492] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef: No bootstrap required, opened a new log
I20260927 03:23:25.889485 8417 ts_tablet_manager.cc:1397] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:25.889500 8415 ts_tablet_manager.cc:1428] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.889498 8416 ts_tablet_manager.cc:1428] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.889617 8417 raft_consensus.cc:348] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.889667 8417 raft_consensus.cc:374] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:25.889678 8417 raft_consensus.cc:729] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5283d7913c4447085236c190fd059ef, State: Initialized, Role: FOLLOWER
I20260927 03:23:25.889710 8417 consensus_queue.cc:260] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.889791 8417 ts_tablet_manager.cc:1428] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:25.889873 8425 tablet_replica.cc:439] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } }
I20260927 03:23:25.889887 8422 tablet_replica.cc:439] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } }
I20260927 03:23:25.889961 8425 tablet_replica.cc:442] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:23:25.889984 8422 tablet_replica.cc:442] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:23:25.890075 8423 tablet_replica.cc:439] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } }
I20260927 03:23:25.890131 8423 tablet_replica.cc:442] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 03:23:25.926954 8388 tablet.cc:2369] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:23:25.944842 8422 raft_consensus.cc:482] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:25.944971 8422 raft_consensus.cc:504] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.945286 8422 leader_election.cc:288] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4aef3b67d7a4d048bb7c4e1a0655d09 (127.0.98.1:44905), aa6a9f51988347d49434496e6e08d2d4 (127.0.98.3:33389)
I20260927 03:23:25.949174 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.945634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38069 (local address 127.0.98.1:44905)
0927 03:23:25.945707 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.945709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.945785 (+ 76us) server_negotiation.cc:407] Connection header received
0927 03:23:25.945830 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.945831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.945838 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.945873 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:25.946019 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.946745 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.948226 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.948845 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.948914 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.948926 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.949058 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.949091 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 03:23:25.949234 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.945607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43239 (local address 127.0.98.3:33389)
0927 03:23:25.945650 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.945652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.945662 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:25.945695 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.945696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.945709 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.945769 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:25.945938 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.946667 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.948356 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.948949 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.949000 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.949008 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.949140 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.949176 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:23:25.949254 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.945503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33389 (local address 127.0.98.2:43239)
0927 03:23:25.945561 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.945574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.945637 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.945783 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.945784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.945785 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:25.945863 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.945865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.946770 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.946771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.948281 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.948283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.949015 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.949016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.949079 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.949082 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.949130 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.949164 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:23:25.949186 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.945503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44905 (local address 127.0.98.2:38069)
0927 03:23:25.945714 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.945723 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.945777 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.945892 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.945892 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.945894 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:25.945957 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.945958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.946853 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.946853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.948137 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.948141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.948914 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.948915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.948981 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.948986 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.949041 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.949086 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":16,"thread_start_us":121,"threads_started":1}
I20260927 03:23:25.949535 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e638def81f754691a7d9f92619d70016" candidate_uuid: "f5283d7913c4447085236c190fd059ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6a9f51988347d49434496e6e08d2d4" is_pre_election: true
I20260927 03:23:25.949636 8341 raft_consensus.cc:2454] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5283d7913c4447085236c190fd059ef in term 0.
I20260927 03:23:25.949728 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e638def81f754691a7d9f92619d70016" candidate_uuid: "f5283d7913c4447085236c190fd059ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" is_pre_election: true
I20260927 03:23:25.949818 8061 raft_consensus.cc:2454] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5283d7913c4447085236c190fd059ef in term 0.
I20260927 03:23:25.949816 8133 leader_election.cc:302] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa6a9f51988347d49434496e6e08d2d4, f5283d7913c4447085236c190fd059ef; no voters:
I20260927 03:23:25.949965 8422 raft_consensus.cc:2781] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:23:25.950012 8422 raft_consensus.cc:482] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:25.950027 8422 raft_consensus.cc:3037] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:25.950675 8422 raft_consensus.cc:504] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.950793 8422 leader_election.cc:288] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [CANDIDATE]: Term 1 election: Requested vote from peers f4aef3b67d7a4d048bb7c4e1a0655d09 (127.0.98.1:44905), aa6a9f51988347d49434496e6e08d2d4 (127.0.98.3:33389)
I20260927 03:23:25.950995 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e638def81f754691a7d9f92619d70016" candidate_uuid: "f5283d7913c4447085236c190fd059ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09"
I20260927 03:23:25.951059 8061 raft_consensus.cc:3037] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:25.951056 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e638def81f754691a7d9f92619d70016" candidate_uuid: "f5283d7913c4447085236c190fd059ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6a9f51988347d49434496e6e08d2d4"
I20260927 03:23:25.951131 8341 raft_consensus.cc:3037] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:25.951695 8061 raft_consensus.cc:2454] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5283d7913c4447085236c190fd059ef in term 1.
I20260927 03:23:25.951812 8341 raft_consensus.cc:2454] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5283d7913c4447085236c190fd059ef in term 1.
I20260927 03:23:25.951887 8132 leader_election.cc:302] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f4aef3b67d7a4d048bb7c4e1a0655d09, f5283d7913c4447085236c190fd059ef; no voters:
I20260927 03:23:25.951978 8422 raft_consensus.cc:2781] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:25.952121 8422 raft_consensus.cc:686] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [term 1 LEADER]: Becoming Leader. State: Replica: f5283d7913c4447085236c190fd059ef, State: Running, Role: LEADER
I20260927 03:23:25.952201 8422 consensus_queue.cc:237] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:25.952556 8456 tablet_replica.cc:439] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef: TxnStatusTablet state changed. Reason: New leader f5283d7913c4447085236c190fd059ef. Latest consensus state: current_term: 1 leader_uuid: "f5283d7913c4447085236c190fd059ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } }
I20260927 03:23:25.952646 8456 tablet_replica.cc:442] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:23:25.952816 8458 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 03:23:25.952926 8458 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 03:23:25.952913 7903 catalog_manager.cc:5495] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef reported cstate change: term changed from 0 to 1, leader changed from <none> to f5283d7913c4447085236c190fd059ef (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f5283d7913c4447085236c190fd059ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } health_report { overall_health: UNKNOWN } } }
I20260927 03:23:25.968992 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.966183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33143 (local address 127.0.0.1:40810)
0927 03:23:25.966283 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.966296 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.966382 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.966564 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.966564 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.966566 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:23:25.966614 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.966616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.967537 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.967538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.968114 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.968116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.968552 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.968553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.968597 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.968613 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:23:25.968867 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:23:25.968871 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.968893 (+ 22us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.968928 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 03:23:25.969010 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.966274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.0.98.2:33143)
0927 03:23:25.966344 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.966346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.966357 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:23:25.966462 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.966464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.966485 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.966552 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:25.966654 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.967448 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.968184 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.968486 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.968549 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.968685 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:25.968815 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:25.968885 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.968901 (+ 16us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.968921 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:23:25.970144 8061 raft_consensus.cc:1261] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Refusing update from remote peer f5283d7913c4447085236c190fd059ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:25.970217 8341 raft_consensus.cc:1261] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Refusing update from remote peer f5283d7913c4447085236c190fd059ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:25.970409 8456 consensus_queue.cc:1035] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:25.970508 8422 consensus_queue.cc:1035] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:25.971089 8423 tablet_replica.cc:439] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09: TxnStatusTablet state changed. Reason: New leader f5283d7913c4447085236c190fd059ef. Latest consensus state: current_term: 1 leader_uuid: "f5283d7913c4447085236c190fd059ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } }
I20260927 03:23:25.971179 8423 tablet_replica.cc:442] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:23:25.971161 8425 tablet_replica.cc:439] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4: TxnStatusTablet state changed. Reason: New leader f5283d7913c4447085236c190fd059ef. Latest consensus state: current_term: 1 leader_uuid: "f5283d7913c4447085236c190fd059ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } }
I20260927 03:23:25.971231 8425 tablet_replica.cc:442] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:23:25.972185 8456 tablet_replica.cc:439] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5283d7913c4447085236c190fd059ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } }
I20260927 03:23:25.972213 8422 tablet_replica.cc:439] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5283d7913c4447085236c190fd059ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } }
I20260927 03:23:25.972273 8456 tablet_replica.cc:442] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:23:25.972289 8422 tablet_replica.cc:442] T e638def81f754691a7d9f92619d70016 P f5283d7913c4447085236c190fd059ef: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:23:25.972863 8428 mvcc.cc:205] Tried to move back new op lower bound from 7333803646851883008 to 7333803646780973056. Current Snapshot: MvccSnapshot[applied={T|T < 7333803646851883008}]
I20260927 03:23:25.972930 8423 tablet_replica.cc:439] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5283d7913c4447085236c190fd059ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } }
I20260927 03:23:25.972967 8425 tablet_replica.cc:439] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5283d7913c4447085236c190fd059ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } } }
I20260927 03:23:25.973006 8423 tablet_replica.cc:442] T e638def81f754691a7d9f92619d70016 P f4aef3b67d7a4d048bb7c4e1a0655d09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:23:25.973023 8425 tablet_replica.cc:442] T e638def81f754691a7d9f92619d70016 P aa6a9f51988347d49434496e6e08d2d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:23:25.977022 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.974111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.0.98.1:44905)
0927 03:23:25.974262 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.974264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.974271 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:23:25.974306 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.974306 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.974327 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.974397 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:25.974649 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.975476 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.976192 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.976489 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.976555 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.976694 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:25.976828 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:25.976900 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.976917 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.976944 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260927 03:23:25.983038 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.979624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46761 (local address 127.0.98.2:33143)
0927 03:23:25.979670 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.979672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.979742 (+ 70us) server_negotiation.cc:407] Connection header received
0927 03:23:25.979803 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.979804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.979812 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.979851 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:25.980043 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.980657 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.982179 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.982385 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.982453 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.982789 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:25.982848 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:25.982892 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.982960 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.982979 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:23:25.983104 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.979538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33143 (local address 127.0.98.1:46761)
0927 03:23:25.979677 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.979687 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.979748 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.979863 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.979864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.979866 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:23:25.979949 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.979950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.980731 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.980732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.982079 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.982082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.982623 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.982624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.982693 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.982710 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:23:25.982906 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:23:25.982907 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.982957 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.983000 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260927 03:23:25.985420 8201 raft_consensus.cc:1261] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Refusing update from remote peer f4aef3b67d7a4d048bb7c4e1a0655d09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:25.985570 8341 raft_consensus.cc:1261] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Refusing update from remote peer f4aef3b67d7a4d048bb7c4e1a0655d09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:25.985672 8423 consensus_queue.cc:1035] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:25.985810 8444 consensus_queue.cc:1035] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:25.994796 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.991917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44905 (local address 127.0.98.2:50117)
0927 03:23:25.991978 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:25.991989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:25.992026 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:25.992283 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:25.992284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:25.992285 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:23:25.992342 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.992344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.992947 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.992948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.993917 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:25.993926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.994447 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.994448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:25.994509 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.994522 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:23:25.994675 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:23:25.994676 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:23:25.994703 (+ 27us) client_negotiation.cc:240] Negotiation successful
0927 03:23:25.994739 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 03:23:25.994800 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:25.992095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50117 (local address 127.0.98.1:44905)
0927 03:23:25.992149 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:25.992151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:25.992159 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:25.992179 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:25.992180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:25.992190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:25.992238 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:25.992412 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.992874 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.994073 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:25.994369 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:25.994432 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:25.994592 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:25.994625 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:25.994666 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:25.994707 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 03:23:25.994725 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:23:26.004376 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:26.004499 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:26.005112 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a: Sending DeleteTablet for 3 replicas of tablet 71e286fc43e9485d8ba9045331c562c3
I20260927 03:23:26.005362 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e286fc43e9485d8ba9045331c562c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55806
I20260927 03:23:26.005455 8438 tablet_replica.cc:330] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09: stopping tablet replica
I20260927 03:23:26.005429 8321 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e286fc43e9485d8ba9045331c562c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50010
I20260927 03:23:26.005496 8438 raft_consensus.cc:2229] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:23:26.005524 8437 tablet_replica.cc:330] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4: stopping tablet replica
I20260927 03:23:26.005578 8437 raft_consensus.cc:2229] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:26.005571 8438 raft_consensus.cc:2258] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:26.005614 8437 raft_consensus.cc:2258] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:26.005971 8437 ts_tablet_manager.cc:1905] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:26.006062 8438 ts_tablet_manager.cc:1905] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:26.007570 8437 ts_tablet_manager.cc:1918] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:23:26.007629 8437 log.cc:1196] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/71e286fc43e9485d8ba9045331c562c3
I20260927 03:23:26.007737 8438 ts_tablet_manager.cc:1918] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:23:26.007800 8438 log.cc:1196] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/wal/wals/71e286fc43e9485d8ba9045331c562c3
I20260927 03:23:26.007824 8437 ts_tablet_manager.cc:1939] T 71e286fc43e9485d8ba9045331c562c3 P aa6a9f51988347d49434496e6e08d2d4: Deleting consensus metadata
I20260927 03:23:26.008064 8438 ts_tablet_manager.cc:1939] T 71e286fc43e9485d8ba9045331c562c3 P f4aef3b67d7a4d048bb7c4e1a0655d09: Deleting consensus metadata
I20260927 03:23:26.008760 7888 catalog_manager.cc:4841] TS f4aef3b67d7a4d048bb7c4e1a0655d09 (127.0.98.1:44905): tablet 71e286fc43e9485d8ba9045331c562c3 (table test-table [id=0f0f97d9565a43478252ec96b12901b0]) successfully deleted
I20260927 03:23:26.008767 7889 catalog_manager.cc:4841] TS aa6a9f51988347d49434496e6e08d2d4 (127.0.98.3:33389): tablet 71e286fc43e9485d8ba9045331c562c3 (table test-table [id=0f0f97d9565a43478252ec96b12901b0]) successfully deleted
I20260927 03:23:26.008948 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e286fc43e9485d8ba9045331c562c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40800
I20260927 03:23:26.009034 8439 tablet_replica.cc:330] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef: stopping tablet replica
I20260927 03:23:26.009079 8439 raft_consensus.cc:2229] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:26.009114 8439 raft_consensus.cc:2258] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:26.009388 8439 ts_tablet_manager.cc:1905] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:26.011286 8439 ts_tablet_manager.cc:1918] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:23:26.011355 8439 log.cc:1196] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/71e286fc43e9485d8ba9045331c562c3
I20260927 03:23:26.011554 8439 ts_tablet_manager.cc:1939] T 71e286fc43e9485d8ba9045331c562c3 P f5283d7913c4447085236c190fd059ef: Deleting consensus metadata
I20260927 03:23:26.011996 7888 catalog_manager.cc:4841] TS f5283d7913c4447085236c190fd059ef (127.0.98.2:33143): tablet 71e286fc43e9485d8ba9045331c562c3 (table test-table [id=0f0f97d9565a43478252ec96b12901b0]) successfully deleted
Sep 27 03:23:26 dist-test-slave-gw2v krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:23:26 dist-test-slave-gw2v krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:26 dist-test-slave-gw2v krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:23:26.045308 7972 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:26.038270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54248 (local address 127.0.98.62:39957)
0927 03:23:26.038320 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:26.038322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:26.038480 (+ 158us) server_negotiation.cc:407] Connection header received
0927 03:23:26.038561 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:26.038562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:26.038574 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:26.038651 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:26.039455 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:26.040218 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:26.041333 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:26.041621 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:26.041702 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:26.045238 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54248: BlockingRecv error: recv got EOF from 127.0.0.1:54248 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 27 03:23:26 dist-test-slave-gw2v krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:23:26.055474 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:26.046379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54254 (local address 127.0.98.62:39957)
0927 03:23:26.046436 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:26.046437 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:26.046674 (+ 237us) server_negotiation.cc:407] Connection header received
0927 03:23:26.046731 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:26.046732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:26.046743 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:26.046806 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:26.047515 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:26.048272 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:26.049298 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:26.049581 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:26.049624 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:26.052426 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:26.052428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:26.052428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:26.052436 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:26.054095 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:26.054511 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:26.054512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:26.054512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:26.054617 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:26.054888 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:26.054889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:26.054890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:26.055052 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:26.055151 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:26.055287 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 03:23:26.055376 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:23:26.057590 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:23:26.057792 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:23:26.059185 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 0e67846651e24698930e019c5e3154f6 (DEFAULT_TABLE table=test-table [id=47a48565713544fbb5e7682db8369cd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:26.059239 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 0e67846651e24698930e019c5e3154f6 (DEFAULT_TABLE table=test-table [id=47a48565713544fbb5e7682db8369cd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:26.059320 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e67846651e24698930e019c5e3154f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:26.059365 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e67846651e24698930e019c5e3154f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:26.060600 8415 tablet_bootstrap.cc:492] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09: Bootstrap starting.
I20260927 03:23:26.060751 8321 tablet_service.cc:1467] Processing CreateTablet for tablet 0e67846651e24698930e019c5e3154f6 (DEFAULT_TABLE table=test-table [id=47a48565713544fbb5e7682db8369cd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:26.060878 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e67846651e24698930e019c5e3154f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:26.061007 8415 tablet_bootstrap.cc:654] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:26.062168 8416 tablet_bootstrap.cc:492] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4: Bootstrap starting.
I20260927 03:23:26.062178 8417 tablet_bootstrap.cc:492] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef: Bootstrap starting.
I20260927 03:23:26.062489 8416 tablet_bootstrap.cc:654] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:26.062511 8417 tablet_bootstrap.cc:654] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:26.063107 8416 tablet_bootstrap.cc:492] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4: No bootstrap required, opened a new log
I20260927 03:23:26.063153 8416 ts_tablet_manager.cc:1397] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:26.063254 8416 raft_consensus.cc:348] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.063320 8416 raft_consensus.cc:374] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:26.063338 8416 raft_consensus.cc:729] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6a9f51988347d49434496e6e08d2d4, State: Initialized, Role: FOLLOWER
I20260927 03:23:26.063378 8416 consensus_queue.cc:260] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.063459 8416 ts_tablet_manager.cc:1428] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:26.063517 8417 tablet_bootstrap.cc:492] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef: No bootstrap required, opened a new log
I20260927 03:23:26.063560 8417 ts_tablet_manager.cc:1397] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:26.063702 8417 raft_consensus.cc:348] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.063766 8417 raft_consensus.cc:374] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:26.063795 8417 raft_consensus.cc:729] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5283d7913c4447085236c190fd059ef, State: Initialized, Role: FOLLOWER
I20260927 03:23:26.063834 8417 consensus_queue.cc:260] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.063906 8417 ts_tablet_manager.cc:1428] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:26.065210 8415 tablet_bootstrap.cc:492] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09: No bootstrap required, opened a new log
I20260927 03:23:26.065264 8415 ts_tablet_manager.cc:1397] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.000s
I20260927 03:23:26.065383 8415 raft_consensus.cc:348] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.065452 8415 raft_consensus.cc:374] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:26.065470 8415 raft_consensus.cc:729] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4aef3b67d7a4d048bb7c4e1a0655d09, State: Initialized, Role: FOLLOWER
I20260927 03:23:26.065508 8415 consensus_queue.cc:260] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.065582 8415 ts_tablet_manager.cc:1428] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:26.086953 8422 raft_consensus.cc:482] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:26.087073 8422 raft_consensus.cc:504] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.087249 8422 leader_election.cc:288] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4aef3b67d7a4d048bb7c4e1a0655d09 (127.0.98.1:44905), aa6a9f51988347d49434496e6e08d2d4 (127.0.98.3:33389)
I20260927 03:23:26.087563 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e67846651e24698930e019c5e3154f6" candidate_uuid: "f5283d7913c4447085236c190fd059ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" is_pre_election: true
I20260927 03:23:26.087600 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e67846651e24698930e019c5e3154f6" candidate_uuid: "f5283d7913c4447085236c190fd059ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6a9f51988347d49434496e6e08d2d4" is_pre_election: true
I20260927 03:23:26.087675 8061 raft_consensus.cc:2454] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5283d7913c4447085236c190fd059ef in term 0.
I20260927 03:23:26.087687 8341 raft_consensus.cc:2454] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5283d7913c4447085236c190fd059ef in term 0.
I20260927 03:23:26.087852 8133 leader_election.cc:302] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: aa6a9f51988347d49434496e6e08d2d4, f4aef3b67d7a4d048bb7c4e1a0655d09, f5283d7913c4447085236c190fd059ef; no voters:
I20260927 03:23:26.088002 8422 raft_consensus.cc:2781] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:23:26.088044 8422 raft_consensus.cc:482] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:26.088066 8422 raft_consensus.cc:3037] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:26.088821 8422 raft_consensus.cc:504] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.088961 8422 leader_election.cc:288] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [CANDIDATE]: Term 1 election: Requested vote from peers f4aef3b67d7a4d048bb7c4e1a0655d09 (127.0.98.1:44905), aa6a9f51988347d49434496e6e08d2d4 (127.0.98.3:33389)
I20260927 03:23:26.089179 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e67846651e24698930e019c5e3154f6" candidate_uuid: "f5283d7913c4447085236c190fd059ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6a9f51988347d49434496e6e08d2d4"
I20260927 03:23:26.089206 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e67846651e24698930e019c5e3154f6" candidate_uuid: "f5283d7913c4447085236c190fd059ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09"
I20260927 03:23:26.089268 8341 raft_consensus.cc:3037] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:26.089272 8061 raft_consensus.cc:3037] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:26.090024 8341 raft_consensus.cc:2454] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5283d7913c4447085236c190fd059ef in term 1.
I20260927 03:23:26.090025 8061 raft_consensus.cc:2454] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5283d7913c4447085236c190fd059ef in term 1.
I20260927 03:23:26.090215 8133 leader_election.cc:302] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa6a9f51988347d49434496e6e08d2d4, f5283d7913c4447085236c190fd059ef; no voters:
I20260927 03:23:26.090337 8422 raft_consensus.cc:2781] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:26.090385 8422 raft_consensus.cc:686] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [term 1 LEADER]: Becoming Leader. State: Replica: f5283d7913c4447085236c190fd059ef, State: Running, Role: LEADER
I20260927 03:23:26.090436 8422 consensus_queue.cc:237] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.091024 7904 catalog_manager.cc:5495] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef reported cstate change: term changed from 0 to 1, leader changed from <none> to f5283d7913c4447085236c190fd059ef (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f5283d7913c4447085236c190fd059ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } health_report { overall_health: UNKNOWN } } }
I20260927 03:23:26.097779 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:26.095294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.0.98.2:33143)
0927 03:23:26.095333 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:26.095335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:26.095343 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:26.095407 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:26.095408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:26.095418 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:26.095462 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:26.095646 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:26.096190 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:26.097005 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:26.097274 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:26.097331 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:26.097517 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:26.097553 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:26.097597 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:26.097656 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 03:23:26.097677 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:23:26.098606 8341 raft_consensus.cc:1261] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Refusing update from remote peer f5283d7913c4447085236c190fd059ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:26.098625 8061 raft_consensus.cc:1261] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Refusing update from remote peer f5283d7913c4447085236c190fd059ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:26.098841 8456 consensus_queue.cc:1035] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:26.099077 8456 consensus_queue.cc:1035] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:26.102417 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:26.102514 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:26.103238 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a: Sending DeleteTablet for 3 replicas of tablet 0e67846651e24698930e019c5e3154f6
I20260927 03:23:26.103466 8321 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e67846651e24698930e019c5e3154f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50010
I20260927 03:23:26.103534 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e67846651e24698930e019c5e3154f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40800
I20260927 03:23:26.103559 8437 tablet_replica.cc:330] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4: stopping tablet replica
I20260927 03:23:26.103608 8437 raft_consensus.cc:2229] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:26.103617 8439 tablet_replica.cc:330] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef: stopping tablet replica
I20260927 03:23:26.103645 8437 raft_consensus.cc:2258] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:26.103652 8439 raft_consensus.cc:2229] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:23:26.103755 8439 raft_consensus.cc:2258] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:26.103899 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e67846651e24698930e019c5e3154f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55806
I20260927 03:23:26.103982 8438 tablet_replica.cc:330] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09: stopping tablet replica
I20260927 03:23:26.104027 8438 raft_consensus.cc:2229] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:26.104061 8438 pending_rounds.cc:62] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09: Trying to abort 2 pending ops.
I20260927 03:23:26.104080 8438 pending_rounds.cc:68] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333803647378419712 op_type: WRITE_OP write_request { tablet_id: "0e67846651e24698930e019c5e3154f6" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\332\232\342\325\006\"8\n\ntest-admin\022*\n 47a48565713544fbb5e7682db8369cd8\020\001\030\001 \001(\001" signature: "<redacted>""n\016\236#\037\371!%\035\235kk\035\303\363\023\247\325\323\206\rW\253\244\003\341\003\331n7\355\204\000\005\317\024\034\217\253\ri~\201c\256\306\347r\237\275\323\322\333`\224#x+\210J2\263GI" signing_key_seq_num: 0 } } request_id { client_id: "b7066fdd6a3c45f2abd86ba998fc22e3" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260927 03:23:26.104269 8438 pending_rounds.cc:68] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333803647346937856 op_type: NO_OP noop_request { }
I20260927 03:23:26.104310 8438 raft_consensus.cc:2866] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260927 03:23:26.104333 8438 raft_consensus.cc:2258] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:26.104338 8437 ts_tablet_manager.cc:1905] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:26.104584 8439 ts_tablet_manager.cc:1905] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:26.106021 8437 ts_tablet_manager.cc:1918] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:26.106072 8437 log.cc:1196] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/0e67846651e24698930e019c5e3154f6
I20260927 03:23:26.106262 8437 ts_tablet_manager.cc:1939] T 0e67846651e24698930e019c5e3154f6 P aa6a9f51988347d49434496e6e08d2d4: Deleting consensus metadata
I20260927 03:23:26.106240 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:23:26.106386 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:23:26.106545 7889 catalog_manager.cc:4841] TS aa6a9f51988347d49434496e6e08d2d4 (127.0.98.3:33389): tablet 0e67846651e24698930e019c5e3154f6 (table test-table [id=47a48565713544fbb5e7682db8369cd8]) successfully deleted
I20260927 03:23:26.106913 8439 ts_tablet_manager.cc:1918] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:26.106966 8439 log.cc:1196] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/0e67846651e24698930e019c5e3154f6
I20260927 03:23:26.107143 8439 ts_tablet_manager.cc:1939] T 0e67846651e24698930e019c5e3154f6 P f5283d7913c4447085236c190fd059ef: Deleting consensus metadata
I20260927 03:23:26.107744 8321 tablet_service.cc:1467] Processing CreateTablet for tablet 03ccfdbb9cf44f2a8383ae700f7ff75f (DEFAULT_TABLE table=test-table [id=4c58a7da2859492db915a7eda06bf7cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:26.107780 7888 catalog_manager.cc:4841] TS f5283d7913c4447085236c190fd059ef (127.0.98.2:33143): tablet 0e67846651e24698930e019c5e3154f6 (table test-table [id=47a48565713544fbb5e7682db8369cd8]) successfully deleted
I20260927 03:23:26.107847 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03ccfdbb9cf44f2a8383ae700f7ff75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:26.107931 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 03ccfdbb9cf44f2a8383ae700f7ff75f (DEFAULT_TABLE table=test-table [id=4c58a7da2859492db915a7eda06bf7cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:26.108006 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03ccfdbb9cf44f2a8383ae700f7ff75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:26.108927 8416 tablet_bootstrap.cc:492] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4: Bootstrap starting.
I20260927 03:23:26.109062 8417 tablet_bootstrap.cc:492] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef: Bootstrap starting.
I20260927 03:23:26.109190 8416 tablet_bootstrap.cc:654] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:26.109329 8417 tablet_bootstrap.cc:654] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:26.109665 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 03ccfdbb9cf44f2a8383ae700f7ff75f (DEFAULT_TABLE table=test-table [id=4c58a7da2859492db915a7eda06bf7cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:26.109776 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03ccfdbb9cf44f2a8383ae700f7ff75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:26.109956 8416 tablet_bootstrap.cc:492] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4: No bootstrap required, opened a new log
I20260927 03:23:26.109993 8416 ts_tablet_manager.cc:1397] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:26.110081 8416 raft_consensus.cc:348] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.110132 8416 raft_consensus.cc:374] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:26.110147 8416 raft_consensus.cc:729] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6a9f51988347d49434496e6e08d2d4, State: Initialized, Role: FOLLOWER
I20260927 03:23:26.110180 8416 consensus_queue.cc:260] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.110258 8416 ts_tablet_manager.cc:1428] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:26.111306 8415 tablet_bootstrap.cc:492] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09: Bootstrap starting.
I20260927 03:23:26.111627 8438 ts_tablet_manager.cc:1905] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:26.111680 8415 tablet_bootstrap.cc:654] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:26.112241 8415 tablet_bootstrap.cc:492] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09: No bootstrap required, opened a new log
I20260927 03:23:26.112283 8415 ts_tablet_manager.cc:1397] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:23:26.112445 8415 raft_consensus.cc:348] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.112506 8415 raft_consensus.cc:374] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:26.112524 8415 raft_consensus.cc:729] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4aef3b67d7a4d048bb7c4e1a0655d09, State: Initialized, Role: FOLLOWER
I20260927 03:23:26.112560 8415 consensus_queue.cc:260] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.112635 8415 ts_tablet_manager.cc:1428] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:26.113142 8438 ts_tablet_manager.cc:1918] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:26.113190 8438 log.cc:1196] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-0/wal/wals/0e67846651e24698930e019c5e3154f6
I20260927 03:23:26.113390 8438 ts_tablet_manager.cc:1939] T 0e67846651e24698930e019c5e3154f6 P f4aef3b67d7a4d048bb7c4e1a0655d09: Deleting consensus metadata
I20260927 03:23:26.113668 7888 catalog_manager.cc:4841] TS f4aef3b67d7a4d048bb7c4e1a0655d09 (127.0.98.1:44905): tablet 0e67846651e24698930e019c5e3154f6 (table test-table [id=47a48565713544fbb5e7682db8369cd8]) successfully deleted
I20260927 03:23:26.113979 8417 tablet_bootstrap.cc:492] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef: No bootstrap required, opened a new log
I20260927 03:23:26.114034 8417 ts_tablet_manager.cc:1397] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.000s
I20260927 03:23:26.114135 8417 raft_consensus.cc:348] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.114199 8417 raft_consensus.cc:374] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:26.114217 8417 raft_consensus.cc:729] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5283d7913c4447085236c190fd059ef, State: Initialized, Role: FOLLOWER
I20260927 03:23:26.114256 8417 consensus_queue.cc:260] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.114348 8417 ts_tablet_manager.cc:1428] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:26.210940 8422 raft_consensus.cc:482] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:26.211041 8422 raft_consensus.cc:504] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.211179 8422 leader_election.cc:288] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4aef3b67d7a4d048bb7c4e1a0655d09 (127.0.98.1:44905), aa6a9f51988347d49434496e6e08d2d4 (127.0.98.3:33389)
I20260927 03:23:26.211410 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03ccfdbb9cf44f2a8383ae700f7ff75f" candidate_uuid: "f5283d7913c4447085236c190fd059ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6a9f51988347d49434496e6e08d2d4" is_pre_election: true
I20260927 03:23:26.211433 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03ccfdbb9cf44f2a8383ae700f7ff75f" candidate_uuid: "f5283d7913c4447085236c190fd059ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" is_pre_election: true
I20260927 03:23:26.211503 8341 raft_consensus.cc:2454] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5283d7913c4447085236c190fd059ef in term 0.
I20260927 03:23:26.211539 8061 raft_consensus.cc:2454] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5283d7913c4447085236c190fd059ef in term 0.
I20260927 03:23:26.211656 8133 leader_election.cc:302] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa6a9f51988347d49434496e6e08d2d4, f5283d7913c4447085236c190fd059ef; no voters:
I20260927 03:23:26.211798 8422 raft_consensus.cc:2781] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:23:26.211853 8422 raft_consensus.cc:482] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:26.211872 8422 raft_consensus.cc:3037] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:26.212677 8422 raft_consensus.cc:504] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.212826 8422 leader_election.cc:288] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [CANDIDATE]: Term 1 election: Requested vote from peers f4aef3b67d7a4d048bb7c4e1a0655d09 (127.0.98.1:44905), aa6a9f51988347d49434496e6e08d2d4 (127.0.98.3:33389)
I20260927 03:23:26.213012 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03ccfdbb9cf44f2a8383ae700f7ff75f" candidate_uuid: "f5283d7913c4447085236c190fd059ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09"
I20260927 03:23:26.213012 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03ccfdbb9cf44f2a8383ae700f7ff75f" candidate_uuid: "f5283d7913c4447085236c190fd059ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6a9f51988347d49434496e6e08d2d4"
I20260927 03:23:26.213088 8061 raft_consensus.cc:3037] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:26.213088 8341 raft_consensus.cc:3037] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:26.213729 8341 raft_consensus.cc:2454] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5283d7913c4447085236c190fd059ef in term 1.
I20260927 03:23:26.213802 8061 raft_consensus.cc:2454] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5283d7913c4447085236c190fd059ef in term 1.
I20260927 03:23:26.213955 8132 leader_election.cc:302] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f4aef3b67d7a4d048bb7c4e1a0655d09, f5283d7913c4447085236c190fd059ef; no voters:
I20260927 03:23:26.214083 8422 raft_consensus.cc:2781] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:26.214126 8422 raft_consensus.cc:686] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [term 1 LEADER]: Becoming Leader. State: Replica: f5283d7913c4447085236c190fd059ef, State: Running, Role: LEADER
I20260927 03:23:26.214174 8422 consensus_queue.cc:237] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } }
I20260927 03:23:26.214764 7904 catalog_manager.cc:5495] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef reported cstate change: term changed from 0 to 1, leader changed from <none> to f5283d7913c4447085236c190fd059ef (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f5283d7913c4447085236c190fd059ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5283d7913c4447085236c190fd059ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 } health_report { overall_health: UNKNOWN } } }
I20260927 03:23:26.237794 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:26.235417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51009 (local address 127.0.98.2:33143)
0927 03:23:26.235453 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:26.235455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:26.235465 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:26.235522 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:26.235523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:26.235533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:26.235586 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:26.235710 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:26.236408 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:26.237097 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:26.237345 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:26.237418 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:26.237557 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:26.237586 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:26.237625 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:26.237671 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 03:23:26.237687 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:23:26.237794 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:26.235360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33143 (local address 127.0.98.2:51009)
0927 03:23:26.235411 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:26.235422 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:26.235476 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:26.235598 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:26.235599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:26.235600 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:23:26.235647 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:26.235649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:26.236487 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:26.236488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:26.237024 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:26.237026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:26.237433 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:26.237434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:26.237483 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:26.237488 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:23:26.237631 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:23:26.237632 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:23:26.237667 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 03:23:26.237706 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 03:23:26.239791 8341 raft_consensus.cc:1261] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Refusing update from remote peer f5283d7913c4447085236c190fd059ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:26.239794 8061 raft_consensus.cc:1261] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Refusing update from remote peer f5283d7913c4447085236c190fd059ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:26.239979 8456 consensus_queue.cc:1035] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa6a9f51988347d49434496e6e08d2d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:26.240082 8422 consensus_queue.cc:1035] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4aef3b67d7a4d048bb7c4e1a0655d09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:26.249698 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:26.249795 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:26.250315 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 663c2a7162da4463b85aa8e2064b2f8a: Sending DeleteTablet for 3 replicas of tablet 03ccfdbb9cf44f2a8383ae700f7ff75f
I20260927 03:23:26.250540 8321 tablet_service.cc:1514] Processing DeleteTablet for tablet 03ccfdbb9cf44f2a8383ae700f7ff75f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50010
I20260927 03:23:26.250545 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 03ccfdbb9cf44f2a8383ae700f7ff75f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55806
I20260927 03:23:26.250582 8180 tablet_service.cc:1514] Processing DeleteTablet for tablet 03ccfdbb9cf44f2a8383ae700f7ff75f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40800
I20260927 03:23:26.250649 8438 tablet_replica.cc:330] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09: stopping tablet replica
I20260927 03:23:26.250671 8437 tablet_replica.cc:330] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4: stopping tablet replica
I20260927 03:23:26.250684 8439 tablet_replica.cc:330] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef: stopping tablet replica
I20260927 03:23:26.250694 8438 raft_consensus.cc:2229] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:26.250717 8437 raft_consensus.cc:2229] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:26.250742 8438 raft_consensus.cc:2258] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f4aef3b67d7a4d048bb7c4e1a0655d09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:26.250743 8439 raft_consensus.cc:2229] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:23:26.250777 8437 raft_consensus.cc:2258] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:26.250806 8439 raft_consensus.cc:2258] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:26.251117 8439 ts_tablet_manager.cc:1905] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:26.251195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 7976
I20260927 03:23:26.251214 8437 ts_tablet_manager.cc:1905] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:26.252835 8437 ts_tablet_manager.cc:1918] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:23:26.252882 8437 log.cc:1196] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/03ccfdbb9cf44f2a8383ae700f7ff75f
I20260927 03:23:26.252915 8439 ts_tablet_manager.cc:1918] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:23:26.252964 8439 log.cc:1196] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/03ccfdbb9cf44f2a8383ae700f7ff75f
I20260927 03:23:26.253124 8437 ts_tablet_manager.cc:1939] T 03ccfdbb9cf44f2a8383ae700f7ff75f P aa6a9f51988347d49434496e6e08d2d4: Deleting consensus metadata
I20260927 03:23:26.253153 8439 ts_tablet_manager.cc:1939] T 03ccfdbb9cf44f2a8383ae700f7ff75f P f5283d7913c4447085236c190fd059ef: Deleting consensus metadata
I20260927 03:23:26.253422 7889 catalog_manager.cc:4841] TS aa6a9f51988347d49434496e6e08d2d4 (127.0.98.3:33389): tablet 03ccfdbb9cf44f2a8383ae700f7ff75f (table test-table [id=4c58a7da2859492db915a7eda06bf7cd]) successfully deleted
I20260927 03:23:26.253436 7888 catalog_manager.cc:4841] TS f5283d7913c4447085236c190fd059ef (127.0.98.2:33143): tablet 03ccfdbb9cf44f2a8383ae700f7ff75f (table test-table [id=4c58a7da2859492db915a7eda06bf7cd]) successfully deleted
W20260927 03:23:26.257218 7888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44905 (error 108)
I20260927 03:23:26.257383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 8116
I20260927 03:23:26.258723 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:26.258544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44905 (local address 127.0.0.1:55830)
0927 03:23:26.258643 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:26.258687 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":39,"threads_started":1}
W20260927 03:23:26.258875 7888 catalog_manager.cc:4584] TS f4aef3b67d7a4d048bb7c4e1a0655d09 (127.0.98.1:44905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03ccfdbb9cf44f2a8383ae700f7ff75f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44905: connect: Connection refused (error 111)
I20260927 03:23:26.263052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 8256
I20260927 03:23:26.268805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 7872
2026-09-27T03:23:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2305 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:23:26 dist-test-slave-gw2v krb5kdc[8503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:23:26 dist-test-slave-gw2v krb5kdc[8503](info): set up 2 sockets
Sep 27 03:23:26 dist-test-slave-gw2v krb5kdc[8503](info): commencing operation
krb5kdc: starting...
W20260927 03:23:27.235128 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:23:27 dist-test-slave-gw2v krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:23:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:23:27.351046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41557
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:27.429291 8516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:27.429502 8516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:27.429524 8516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:27.430616 8516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:27.430646 8516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:27.430658 8516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:27.430670 8516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:27.430680 8516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:27.432133 8516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:27.432336 8516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:27.432511 8516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:27.437131 8521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:27.437119 8524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:27.437258 8522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:27.437312 8516 server_base.cc:1034] running on GCE node
I20260927 03:23:27.437526 8516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:27.437798 8516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:27.438932 8516 hybrid_clock.cc:648] HybridClock initialized: now 1790479407438913 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T03:23:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1179 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:23:27 dist-test-slave-gw2v krb5kdc[8529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:23:27 dist-test-slave-gw2v krb5kdc[8529](info): set up 2 sockets
Sep 27 03:23:27 dist-test-slave-gw2v krb5kdc[8529](info): commencing operation
krb5kdc: starting...
W20260927 03:23:28.395117 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:23:28 dist-test-slave-gw2v krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:23:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:23:28.506464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40217
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:28.585362 8542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:28.585613 8542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:28.585636 8542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:28.586891 8542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:28.586948 8542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:28.586961 8542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:28.586973 8542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:28.586984 8542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:28.588650 8542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:28.588913 8542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:28.589140 8542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:28.594156 8550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:28.594215 8547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:28.594215 8548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:28.594533 8542 server_base.cc:1034] running on GCE node
I20260927 03:23:28.594787 8542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:28.595100 8542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:28.596269 8542 hybrid_clock.cc:648] HybridClock initialized: now 1790479408596248 us; error 33 us; skew 500 ppm
Sep 27 03:23:28 dist-test-slave-gw2v krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:28.598956 8542 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:23:28.599473 8542 webserver.cc:460] Webserver started at http://127.0.98.62:46593/ using document root <none> and password file <none>
I20260927 03:23:28.599689 8542 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:28.599736 8542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:28.599864 8542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:28.600934 8542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "b47406cb23d9484792f648a3e150c46b"
format_stamp: "Formatted at 2026-09-27 03:23:28 on dist-test-slave-gw2v"
server_key: "0ffc8d5791faf4616a6e0afff4625cad"
server_key_iv: "115e790498a18c6573798079252df4d4"
server_key_version: "encryptionkey@0"
I20260927 03:23:28.601270 8542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "b47406cb23d9484792f648a3e150c46b"
format_stamp: "Formatted at 2026-09-27 03:23:28 on dist-test-slave-gw2v"
server_key: "0ffc8d5791faf4616a6e0afff4625cad"
server_key_iv: "115e790498a18c6573798079252df4d4"
server_key_version: "encryptionkey@0"
I20260927 03:23:28.602509 8542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:28.603246 8557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:28.603402 8542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:28.603487 8542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "b47406cb23d9484792f648a3e150c46b"
format_stamp: "Formatted at 2026-09-27 03:23:28 on dist-test-slave-gw2v"
server_key: "0ffc8d5791faf4616a6e0afff4625cad"
server_key_iv: "115e790498a18c6573798079252df4d4"
server_key_version: "encryptionkey@0"
I20260927 03:23:28.603552 8542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:28.611704 8542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:28.613469 8542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:28.613605 8542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:28.617465 8542 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40217
I20260927 03:23:28.617530 8609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40217 every 8 connection(s)
I20260927 03:23:28.617985 8542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:28.618619 8610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:28.620901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 8542
I20260927 03:23:28.620921 8610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b: Bootstrap starting.
I20260927 03:23:28.621078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:28.621258 392 external_mini_cluster.cc:1428] Setting key 25d6a77dbbd0de4b404420d5de487687
I20260927 03:23:28.621513 8610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:28.621824 8610 log.cc:824] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:28.622480 8610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b: No bootstrap required, opened a new log
I20260927 03:23:28.624636 8610 raft_consensus.cc:348] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47406cb23d9484792f648a3e150c46b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } }
I20260927 03:23:28.624814 8610 raft_consensus.cc:374] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:28.624846 8610 raft_consensus.cc:729] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47406cb23d9484792f648a3e150c46b, State: Initialized, Role: FOLLOWER
I20260927 03:23:28.624969 8610 consensus_queue.cc:260] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47406cb23d9484792f648a3e150c46b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } }
I20260927 03:23:28.625036 8610 raft_consensus.cc:388] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:28.625069 8610 raft_consensus.cc:482] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:28.625120 8610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:28.625670 8610 raft_consensus.cc:504] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47406cb23d9484792f648a3e150c46b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } }
I20260927 03:23:28.625769 8610 leader_election.cc:302] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b47406cb23d9484792f648a3e150c46b; no voters:
I20260927 03:23:28.625921 8610 leader_election.cc:288] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:28.626046 8615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:28.626200 8615 raft_consensus.cc:686] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 1 LEADER]: Becoming Leader. State: Replica: b47406cb23d9484792f648a3e150c46b, State: Running, Role: LEADER
I20260927 03:23:28.626315 8615 consensus_queue.cc:237] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47406cb23d9484792f648a3e150c46b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } }
I20260927 03:23:28.626335 8610 sys_catalog.cc:564] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:28.627178 8617 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b47406cb23d9484792f648a3e150c46b. Latest consensus state: current_term: 1 leader_uuid: "b47406cb23d9484792f648a3e150c46b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47406cb23d9484792f648a3e150c46b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } }
I20260927 03:23:28.627429 8616 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b47406cb23d9484792f648a3e150c46b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47406cb23d9484792f648a3e150c46b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } }
I20260927 03:23:28.627488 8617 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:28.627496 8616 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:28.627766 8627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 03:23:28 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:28.628228 8627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:28.629917 8627 catalog_manager.cc:1347] Generated new cluster ID: 2c6a0d38f898448da741f071ec0b4918
I20260927 03:23:28.629961 8627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:23:28.632725 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:28.621753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60698 (local address 127.0.98.62:40217)
0927 03:23:28.621924 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:28.621935 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:28.621967 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:23:28.622454 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:28.622463 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:28.622526 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:28.622665 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:28.623433 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:28.624319 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:28.625331 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:28.625792 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:28.625942 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:28.628786 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:28.628790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:28.628790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:28.628803 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:28.630793 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:28.631182 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:28.631184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:28.631185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:28.631294 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:28.631462 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:28.631463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:28.631463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:28.631774 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:28.631925 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:28.632363 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 03:23:28.632500 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260927 03:23:28.637145 8627 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:28.637641 8627 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:28.641070 8627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b: Generated new TSK 0
I20260927 03:23:28.641256 8627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:23:28.673933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40217
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:28.753885 8636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:28.754097 8636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:28.754122 8636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:28.755304 8636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:28.755342 8636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:28.755366 8636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:23:28.756953 8636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:28.757164 8636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:28.757350 8636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:23:28.762342 8636 server_base.cc:1034] running on GCE node
W20260927 03:23:28.762291 8644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:28.762307 8642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:28.762312 8641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:28.763784 8636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:28.763998 8636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:28.765192 8636 hybrid_clock.cc:648] HybridClock initialized: now 1790479408765166 us; error 42 us; skew 500 ppm
Sep 27 03:23:28 dist-test-slave-gw2v krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:28.767870 8636 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:23:28.768386 8636 webserver.cc:460] Webserver started at http://127.0.98.1:43041/ using document root <none> and password file <none>
I20260927 03:23:28.768603 8636 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:28.768647 8636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:28.768772 8636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:28.769832 8636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "4cccd1233c1a47029611fbaa3f8b7efc"
format_stamp: "Formatted at 2026-09-27 03:23:28 on dist-test-slave-gw2v"
server_key: "7998b867e7f48e1f60d20448ad6b73bc"
server_key_iv: "b66ce1adef0dab30ee93723cfdbe61ca"
server_key_version: "encryptionkey@0"
I20260927 03:23:28.770128 8636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4cccd1233c1a47029611fbaa3f8b7efc"
format_stamp: "Formatted at 2026-09-27 03:23:28 on dist-test-slave-gw2v"
server_key: "7998b867e7f48e1f60d20448ad6b73bc"
server_key_iv: "b66ce1adef0dab30ee93723cfdbe61ca"
server_key_version: "encryptionkey@0"
I20260927 03:23:28.771391 8636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:28.772033 8651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:28.772188 8636 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:28.772270 8636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "4cccd1233c1a47029611fbaa3f8b7efc"
format_stamp: "Formatted at 2026-09-27 03:23:28 on dist-test-slave-gw2v"
server_key: "7998b867e7f48e1f60d20448ad6b73bc"
server_key_iv: "b66ce1adef0dab30ee93723cfdbe61ca"
server_key_version: "encryptionkey@0"
I20260927 03:23:28.772330 8636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:28.791079 8636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:28.793473 8636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:28.793619 8636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:28.793840 8636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:28.794162 8636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:28.794198 8636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:28.794226 8636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:28.794243 8636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:28.799779 8636 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35039
I20260927 03:23:28.799837 8764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35039 every 8 connection(s)
I20260927 03:23:28.800132 8636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:23:28 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:28.808866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 8636
I20260927 03:23:28.809051 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:23:28.809063 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:28.800792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38823 (local address 127.0.98.62:40217)
0927 03:23:28.800835 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:28.800837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:28.801040 (+ 203us) server_negotiation.cc:407] Connection header received
0927 03:23:28.801962 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:28.801964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:28.801976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:28.802019 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:28.802657 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:28.803235 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:28.804211 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:28.804425 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:28.804489 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:28.806652 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:28.806654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:28.806655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:28.806663 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:28.807926 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:28.808289 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:28.808290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:28.808290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:28.808339 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:28.808511 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:28.808511 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:28.808511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:28.808605 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:28.808648 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:28.808888 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 03:23:28.808975 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:23:28.809237 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:28.800764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:38823)
0927 03:23:28.800914 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:28.800950 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:28.801856 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:28.802117 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:28.802120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:28.802375 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:28.802567 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:28.802571 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:28.803315 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:28.803316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:28.804131 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:28.804133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:28.804492 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:28.804493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:28.804623 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:28.805090 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:28.805098 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:28.806559 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:28.808004 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:28.808009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:28.808011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:28.808234 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:28.808384 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:28.808385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:28.808385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:28.808472 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:28.808655 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:28.808660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:28.808774 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 03:23:28.808855 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 03:23:28.808996 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260927 03:23:28.809252 392 external_mini_cluster.cc:1428] Setting key 53b2924dcddea4354af82e6287415996
I20260927 03:23:28.809674 8765 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40217
I20260927 03:23:28.809767 8765 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:28.809945 8765 heartbeater.cc:511] Master 127.0.98.62:40217 requested a full tablet report, sending...
I20260927 03:23:28.810467 8574 ts_manager.cc:194] Registered new tserver with Master: 4cccd1233c1a47029611fbaa3f8b7efc (127.0.98.1:35039)
I20260927 03:23:28.811314 8574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38823
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:23:28.833281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40217
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:28.914812 8770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:28.915119 8770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:28.915154 8770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:28.916451 8770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:28.916502 8770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:28.916539 8770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:23:28.918147 8770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:28.918401 8770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:28.918622 8770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:28.923597 8775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:28.923563 8776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:28.923581 8778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:28.923656 8770 server_base.cc:1034] running on GCE node
I20260927 03:23:28.923985 8770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:28.924300 8770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:28.925457 8770 hybrid_clock.cc:648] HybridClock initialized: now 1790479408925444 us; error 30 us; skew 500 ppm
Sep 27 03:23:28 dist-test-slave-gw2v krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:28.931502 8770 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:23:28.931974 8770 webserver.cc:460] Webserver started at http://127.0.98.2:43001/ using document root <none> and password file <none>
I20260927 03:23:28.932240 8770 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:28.932294 8770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:28.932418 8770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:28.933451 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "4b4680ce500e4cc78708967176db2e86"
format_stamp: "Formatted at 2026-09-27 03:23:28 on dist-test-slave-gw2v"
server_key: "2887a67ef032cd481ca5cb7117230d9d"
server_key_iv: "a3660dbb1e32b4cb4828d81663674676"
server_key_version: "encryptionkey@0"
I20260927 03:23:28.933774 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4b4680ce500e4cc78708967176db2e86"
format_stamp: "Formatted at 2026-09-27 03:23:28 on dist-test-slave-gw2v"
server_key: "2887a67ef032cd481ca5cb7117230d9d"
server_key_iv: "a3660dbb1e32b4cb4828d81663674676"
server_key_version: "encryptionkey@0"
I20260927 03:23:28.935130 8770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:28.935871 8785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:28.936115 8770 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:28.936208 8770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "4b4680ce500e4cc78708967176db2e86"
format_stamp: "Formatted at 2026-09-27 03:23:28 on dist-test-slave-gw2v"
server_key: "2887a67ef032cd481ca5cb7117230d9d"
server_key_iv: "a3660dbb1e32b4cb4828d81663674676"
server_key_version: "encryptionkey@0"
I20260927 03:23:28.936273 8770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:28.945221 8770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:28.947786 8770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:28.947932 8770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:28.948176 8770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:28.948469 8770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:28.948504 8770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:28.948532 8770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:28.948565 8770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:28.954058 8770 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44373
I20260927 03:23:28.954108 8898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44373 every 8 connection(s)
I20260927 03:23:28.954422 8770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:23:28.957819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 8770
I20260927 03:23:28.957931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:23:28.958120 392 external_mini_cluster.cc:1428] Setting key 02ad8c54da18e762368fe15b3d0927b7
Sep 27 03:23:28 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:28.964293 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:28.955160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43145 (local address 127.0.98.62:40217)
0927 03:23:28.955224 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:28.955226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:28.955381 (+ 155us) server_negotiation.cc:407] Connection header received
0927 03:23:28.956327 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:28.956328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:28.956339 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:28.956401 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:28.957305 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:28.957798 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:28.959239 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:28.959435 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:28.959495 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:28.961723 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:28.961724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:28.961725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:28.961733 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:28.963040 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:28.963407 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:28.963408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:28.963409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:28.963461 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:28.963681 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:28.963682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:28.963682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:28.963792 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:28.963847 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:28.964115 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 03:23:28.964193 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:23:28.964473 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:28.955085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:43145)
0927 03:23:28.955254 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:28.955296 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:28.956204 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:28.956489 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:28.956492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:28.956920 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:28.957217 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:28.957222 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:28.957893 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:28.957895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:28.959143 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:28.959146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:28.959507 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:28.959508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:28.959649 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:28.960166 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:28.960178 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:28.961625 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:28.963114 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:28.963120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:28.963123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:28.963343 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:28.963518 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:28.963519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:28.963519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:28.963627 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:28.963852 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:28.963859 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:28.963995 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 03:23:28.964093 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 03:23:28.964252 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260927 03:23:28.964895 8899 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40217
I20260927 03:23:28.964983 8899 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:28.965157 8899 heartbeater.cc:511] Master 127.0.98.62:40217 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 03:23:28.965572 8574 ts_manager.cc:194] Registered new tserver with Master: 4b4680ce500e4cc78708967176db2e86 (127.0.98.2:44373)
I20260927 03:23:28.965934 8574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:23:28.982751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40217
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:29.063336 8904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:29.063542 8904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:29.063568 8904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:29.064751 8904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:29.064787 8904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:29.064813 8904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:23:29.066293 8904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:29.066490 8904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:29.066689 8904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:29.071444 8910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:29.071486 8912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:29.071491 8909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:29.071622 8904 server_base.cc:1034] running on GCE node
I20260927 03:23:29.071878 8904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:29.072100 8904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:29.073241 8904 hybrid_clock.cc:648] HybridClock initialized: now 1790479409073228 us; error 29 us; skew 500 ppm
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:29.075778 8904 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:23:29.076289 8904 webserver.cc:460] Webserver started at http://127.0.98.3:43245/ using document root <none> and password file <none>
I20260927 03:23:29.076493 8904 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:29.076538 8904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:29.076660 8904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:29.077754 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "fdc59d5be3f94963821f01d8af04dffb"
format_stamp: "Formatted at 2026-09-27 03:23:29 on dist-test-slave-gw2v"
server_key: "f68158e3cb019db5bd52ecdf547d7c4c"
server_key_iv: "62efb45197c65d6f5c4ff60fba5085be"
server_key_version: "encryptionkey@0"
I20260927 03:23:29.078092 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fdc59d5be3f94963821f01d8af04dffb"
format_stamp: "Formatted at 2026-09-27 03:23:29 on dist-test-slave-gw2v"
server_key: "f68158e3cb019db5bd52ecdf547d7c4c"
server_key_iv: "62efb45197c65d6f5c4ff60fba5085be"
server_key_version: "encryptionkey@0"
I20260927 03:23:29.079429 8904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:23:29.080245 8919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:29.080400 8904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:29.080492 8904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "fdc59d5be3f94963821f01d8af04dffb"
format_stamp: "Formatted at 2026-09-27 03:23:29 on dist-test-slave-gw2v"
server_key: "f68158e3cb019db5bd52ecdf547d7c4c"
server_key_iv: "62efb45197c65d6f5c4ff60fba5085be"
server_key_version: "encryptionkey@0"
I20260927 03:23:29.080554 8904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:29.090204 8904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:29.092267 8904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:29.092399 8904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:29.092617 8904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:29.092890 8904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:29.092922 8904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:29.092952 8904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:29.092974 8904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:29.098263 8904 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37699
I20260927 03:23:29.098328 9032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37699 every 8 connection(s)
I20260927 03:23:29.098594 8904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:29.107353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 8904
I20260927 03:23:29.107439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:23:29.107616 392 external_mini_cluster.cc:1428] Setting key dcab72c9e12bb79f9778c6f57e575666
I20260927 03:23:29.107621 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:29.099376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58913 (local address 127.0.98.62:40217)
0927 03:23:29.099433 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:29.099435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:29.099615 (+ 180us) server_negotiation.cc:407] Connection header received
0927 03:23:29.100638 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:29.100639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:29.100652 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:29.100700 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:29.101318 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.101821 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.102785 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.103027 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.103082 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.105094 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:29.105096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:29.105096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:29.105104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:29.106356 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:29.106717 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:29.106719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:29.106719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:29.106771 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:29.106991 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:29.106992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:29.106992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:29.107102 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:29.107156 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:29.107426 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 03:23:29.107534 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:23:29.107784 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:29.099303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:58913)
0927 03:23:29.099496 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:29.099539 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:29.100527 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:29.100781 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:29.100786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:29.101046 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:29.101234 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.101238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.101900 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.101901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.102706 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.102709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.103094 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.103095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.103219 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.103695 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:29.103703 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:29.105005 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:29.106431 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:29.106438 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:29.106440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:29.106652 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:29.106841 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:29.106841 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:29.106842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:29.106937 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:29.107153 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:29.107160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:29.107312 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 03:23:29.107399 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 03:23:29.107567 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":80,"threads_started":1}
I20260927 03:23:29.108225 9033 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40217
I20260927 03:23:29.108326 9033 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:29.108506 9033 heartbeater.cc:511] Master 127.0.98.62:40217 requested a full tablet report, sending...
I20260927 03:23:29.108853 8574 ts_manager.cc:194] Registered new tserver with Master: fdc59d5be3f94963821f01d8af04dffb (127.0.98.3:37699)
I20260927 03:23:29.109197 8574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58913
I20260927 03:23:29.119235 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 03:23:29.185882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 8636
I20260927 03:23:29.190094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 8770
I20260927 03:23:29.193636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 8904
I20260927 03:23:29.197288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 8542
I20260927 03:23:29.201539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40217
--webserver_interface=127.0.98.62
--webserver_port=46593
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:29.280481 9042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:29.280696 9042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:29.280727 9042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:29.281898 9042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:29.281939 9042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:29.281956 9042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:29.281971 9042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:29.281996 9042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:29.283524 9042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:29.283758 9042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:29.283960 9042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:29.288753 9047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:29.288748 9050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:29.288748 9048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:29.288962 9042 server_base.cc:1034] running on GCE node
I20260927 03:23:29.289443 9042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:29.289729 9042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:29.290886 9042 hybrid_clock.cc:648] HybridClock initialized: now 1790479409290871 us; error 27 us; skew 500 ppm
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:29.293190 9042 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:23:29.293612 9042 webserver.cc:460] Webserver started at http://127.0.98.62:46593/ using document root <none> and password file <none>
I20260927 03:23:29.293798 9042 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:29.293846 9042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:29.295202 9042 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:29.295778 9057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:29.295921 9042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:23:29.295998 9042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "b47406cb23d9484792f648a3e150c46b"
format_stamp: "Formatted at 2026-09-27 03:23:28 on dist-test-slave-gw2v"
server_key: "0ffc8d5791faf4616a6e0afff4625cad"
server_key_iv: "115e790498a18c6573798079252df4d4"
server_key_version: "encryptionkey@0"
I20260927 03:23:29.296331 9042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:29.332130 9042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:29.334220 9042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:29.334337 9042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:29.338127 9042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40217
I20260927 03:23:29.338173 9109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40217 every 8 connection(s)
I20260927 03:23:29.338663 9042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:29.339906 9110 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:23:29.340502 9110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b: Bootstrap starting.
I20260927 03:23:29.346040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 9042
I20260927 03:23:29.346307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35039
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40217
--webserver_port=43041
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:23:29.348204 9110 log.cc:824] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:29.349872 9110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:23:29.350116 9110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b: Bootstrap complete.
I20260927 03:23:29.352169 9110 raft_consensus.cc:348] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47406cb23d9484792f648a3e150c46b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } }
I20260927 03:23:29.352396 9110 raft_consensus.cc:729] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47406cb23d9484792f648a3e150c46b, State: Initialized, Role: FOLLOWER
I20260927 03:23:29.352545 9110 consensus_queue.cc:260] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47406cb23d9484792f648a3e150c46b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } }
I20260927 03:23:29.352622 9110 raft_consensus.cc:388] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:29.352671 9110 raft_consensus.cc:482] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:29.352720 9110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:23:29.353679 9110 raft_consensus.cc:504] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47406cb23d9484792f648a3e150c46b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } }
I20260927 03:23:29.353770 9110 leader_election.cc:302] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b47406cb23d9484792f648a3e150c46b; no voters:
I20260927 03:23:29.353928 9110 leader_election.cc:288] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:23:29.353976 9114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:23:29.354079 9114 raft_consensus.cc:686] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [term 2 LEADER]: Becoming Leader. State: Replica: b47406cb23d9484792f648a3e150c46b, State: Running, Role: LEADER
I20260927 03:23:29.354162 9114 consensus_queue.cc:237] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47406cb23d9484792f648a3e150c46b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } }
I20260927 03:23:29.354194 9110 sys_catalog.cc:564] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:29.354420 9115 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b47406cb23d9484792f648a3e150c46b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47406cb23d9484792f648a3e150c46b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } }
I20260927 03:23:29.354527 9115 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:29.354411 9116 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b47406cb23d9484792f648a3e150c46b. Latest consensus state: current_term: 2 leader_uuid: "b47406cb23d9484792f648a3e150c46b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47406cb23d9484792f648a3e150c46b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } }
I20260927 03:23:29.354861 9116 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:29.355000 9119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:29.355468 9119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:29.355584 9119 catalog_manager.cc:1259] Loaded cluster ID: 2c6a0d38f898448da741f071ec0b4918
I20260927 03:23:29.355808 9119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:23:29.356985 9119 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:29.357174 9119 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b47406cb23d9484792f648a3e150c46b: Loaded TSK: 0
I20260927 03:23:29.357389 9119 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:23:29.427032 9112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:29.427253 9112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:29.427285 9112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:29.428512 9112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:29.428557 9112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:29.428596 9112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:23:29.430038 9112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43041
--tserver_master_addrs=127.0.98.62:40217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:29.430281 9112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:29.430516 9112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:29.435480 9139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:29.435488 9137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:29.435534 9136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:29.435794 9112 server_base.cc:1034] running on GCE node
I20260927 03:23:29.435971 9112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:29.436197 9112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:29.437343 9112 hybrid_clock.cc:648] HybridClock initialized: now 1790479409437325 us; error 32 us; skew 500 ppm
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:29.439803 9112 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 03:23:29.440235 9112 webserver.cc:460] Webserver started at http://127.0.98.1:43041/ using document root <none> and password file <none>
I20260927 03:23:29.440475 9112 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:29.440541 9112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:29.441834 9112 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:29.442377 9146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:29.442530 9112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:29.442606 9112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "4cccd1233c1a47029611fbaa3f8b7efc"
format_stamp: "Formatted at 2026-09-27 03:23:28 on dist-test-slave-gw2v"
server_key: "7998b867e7f48e1f60d20448ad6b73bc"
server_key_iv: "b66ce1adef0dab30ee93723cfdbe61ca"
server_key_version: "encryptionkey@0"
I20260927 03:23:29.443019 9112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:29.459920 9112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:29.462028 9112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:29.462158 9112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:29.462363 9112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:29.462656 9112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:29.462687 9112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:29.462715 9112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:29.462739 9112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:29.469286 9112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35039
I20260927 03:23:29.469357 9259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35039 every 8 connection(s)
I20260927 03:23:29.469681 9112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:23:29.470989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 9112
I20260927 03:23:29.471264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44373
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40217
--webserver_port=43001
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:29.478734 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:29.470349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:36347)
0927 03:23:29.470502 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:29.470542 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:29.471083 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:29.471600 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:29.471604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:29.472156 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:29.472351 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.472356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.473421 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.473422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.474637 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.474640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.475443 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.475445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.475604 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.476134 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:29.476144 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:29.478555 (+ 2411us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
W20260927 03:23:29.478936 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:29.470410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36347 (local address 127.0.98.62:40217)
0927 03:23:29.470584 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:29.470601 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:29.470634 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:23:29.471270 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:29.471276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:29.471327 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:29.471458 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:29.472468 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.473324 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.474731 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.475341 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.475457 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.478716 (+ 3259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36347: BlockingRecv error: recv got EOF from 127.0.98.1:36347 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
W20260927 03:23:29.478951 9260 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40217 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40217: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:29.483465 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:29.479167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:33905)
0927 03:23:29.479215 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:29.479280 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:29.479334 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:29.479459 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:29.479459 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:29.479562 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:29.479631 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.479632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.480482 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.480483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.480984 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.480985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.481404 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.481405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.481447 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.481851 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:29.481855 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:29.483419 (+ 1564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 03:23:29.483570 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:29.479256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33905 (local address 127.0.98.62:40217)
0927 03:23:29.479294 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:29.479296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:29.479320 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:23:29.479383 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:29.479384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:29.479396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:29.479453 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:29.479693 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.480406 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.481055 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.481337 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.481402 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.483519 (+ 2117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33905: BlockingRecv error: recv got EOF from 127.0.98.1:33905 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:29.488050 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:29.483712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:58045)
0927 03:23:29.483747 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:29.483757 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:29.483795 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:29.483964 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:29.483965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:29.484057 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:29.484083 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.484084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.484957 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.484958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.485466 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.485467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.485887 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.485887 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.485920 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.486295 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:29.486299 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:29.487984 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":19}
W20260927 03:23:29.488122 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:29.483837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58045 (local address 127.0.98.62:40217)
0927 03:23:29.483870 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:29.483871 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:29.483877 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:23:29.483895 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:29.483896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:29.483905 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:29.483952 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:29.484146 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.484872 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.485530 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.485824 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.485880 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.488063 (+ 2183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58045: BlockingRecv error: recv got EOF from 127.0.98.1:58045 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 03:23:29.488314 9260 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:23:29.554168 9264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:29.554438 9264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:29.554482 9264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:29.555810 9264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:29.555856 9264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:29.555882 9264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:23:29.557367 9264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43001
--tserver_master_addrs=127.0.98.62:40217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:29.557595 9264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:29.557806 9264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:29.562726 9269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:29.562767 9264 server_base.cc:1034] running on GCE node
W20260927 03:23:29.562958 9270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:29.562976 9272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:29.563186 9264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:29.563421 9264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:29.564561 9264 hybrid_clock.cc:648] HybridClock initialized: now 1790479409564548 us; error 28 us; skew 500 ppm
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:29.567018 9264 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 03:23:29.567471 9264 webserver.cc:460] Webserver started at http://127.0.98.2:43001/ using document root <none> and password file <none>
I20260927 03:23:29.567687 9264 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:29.567749 9264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:29.569090 9264 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:29.569674 9279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:29.569834 9264 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:29.569911 9264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "4b4680ce500e4cc78708967176db2e86"
format_stamp: "Formatted at 2026-09-27 03:23:28 on dist-test-slave-gw2v"
server_key: "2887a67ef032cd481ca5cb7117230d9d"
server_key_iv: "a3660dbb1e32b4cb4828d81663674676"
server_key_version: "encryptionkey@0"
I20260927 03:23:29.570288 9264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:29.597400 9264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:29.599476 9264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:29.599606 9264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:29.599814 9264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:29.600066 9264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:29.600095 9264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:29.600124 9264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:29.600140 9264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:29.605366 9264 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44373
I20260927 03:23:29.605413 9392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44373 every 8 connection(s)
I20260927 03:23:29.605739 9264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:23:29.606838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 9264
I20260927 03:23:29.607178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37699
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40217
--webserver_port=43245
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:29.613487 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:29.606417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:58349)
0927 03:23:29.606557 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:29.606591 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:29.607242 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:29.607546 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:29.607550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:29.608246 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:29.608546 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.608554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.609475 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.609475 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.610270 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.610272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.610742 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.610743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.610885 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.611359 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:29.611367 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:29.613328 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
W20260927 03:23:29.613504 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:29.606410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58349 (local address 127.0.98.62:40217)
0927 03:23:29.606470 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:29.606472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:29.606680 (+ 208us) server_negotiation.cc:407] Connection header received
0927 03:23:29.607379 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:29.607380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:29.607395 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:29.607446 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:29.608647 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.609367 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.610352 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.610653 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.610719 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.613439 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58349: BlockingRecv error: recv got EOF from 127.0.98.2:58349 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 03:23:29.613720 9393 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40217 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40217: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:29.617790 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:29.613928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:60647)
0927 03:23:29.613974 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:29.613985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:29.614021 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:29.614131 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:29.614132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:29.614247 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:29.614286 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.614287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.614909 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.614910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.615403 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.615404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.615715 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.615716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.615773 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.616173 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:29.616178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:29.617720 (+ 1542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 03:23:29.617843 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:29.613984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60647 (local address 127.0.98.62:40217)
0927 03:23:29.614028 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:29.614030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:29.614053 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:23:29.614076 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:29.614077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:29.614085 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:29.614124 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:29.614347 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.614848 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.615477 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.615662 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.615713 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.617794 (+ 2081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60647: BlockingRecv error: recv got EOF from 127.0.98.2:60647 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:29.622715 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:29.618009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:39521)
0927 03:23:29.618142 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:29.618151 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:29.618184 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:29.618310 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:29.618311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:29.618440 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:29.618472 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.618473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.619156 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.619156 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.619971 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.619973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.620293 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.620294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.620344 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.620758 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:29.620763 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:29.622638 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
W20260927 03:23:29.622798 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:29.618133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39521 (local address 127.0.98.62:40217)
0927 03:23:29.618169 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:29.618171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:29.618180 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:23:29.618248 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:29.618248 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:29.618256 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:29.618295 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:29.618531 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.619094 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.620056 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.620232 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.620288 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.622736 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39521: BlockingRecv error: recv got EOF from 127.0.98.2:39521 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 03:23:29.622871 9393 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:23:29.691012 9396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:29.691218 9396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:29.691241 9396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:29.692406 9396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:29.692447 9396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:29.692472 9396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:23:29.693841 9396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43245
--tserver_master_addrs=127.0.98.62:40217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:29.694031 9396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:29.694240 9396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:29.698889 9402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:29.698927 9405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:29.699030 9403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:29.699143 9396 server_base.cc:1034] running on GCE node
I20260927 03:23:29.699307 9396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:29.699506 9396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:29.700639 9396 hybrid_clock.cc:648] HybridClock initialized: now 1790479409700623 us; error 31 us; skew 500 ppm
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:29.703011 9396 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 03:23:29.703464 9396 webserver.cc:460] Webserver started at http://127.0.98.3:43245/ using document root <none> and password file <none>
I20260927 03:23:29.703677 9396 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:29.703740 9396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:29.705073 9396 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:29.705597 9412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:29.705750 9396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:29.705821 9396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "fdc59d5be3f94963821f01d8af04dffb"
format_stamp: "Formatted at 2026-09-27 03:23:29 on dist-test-slave-gw2v"
server_key: "f68158e3cb019db5bd52ecdf547d7c4c"
server_key_iv: "62efb45197c65d6f5c4ff60fba5085be"
server_key_version: "encryptionkey@0"
I20260927 03:23:29.706197 9396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:29.715824 9396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:29.717710 9396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:29.717834 9396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:29.718039 9396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:29.718305 9396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:29.718336 9396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:29.718365 9396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:29.718389 9396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:29.724063 9396 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37699
I20260927 03:23:29.724116 9525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37699 every 8 connection(s)
I20260927 03:23:29.724427 9396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:29.731290 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:29.725047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:50147)
0927 03:23:29.725187 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:29.725226 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:29.725681 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:29.725869 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:29.725872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:29.726393 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:29.726583 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.726587 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.727328 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.727329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.728136 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.728138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.728484 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.728484 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.728578 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.728954 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:29.728961 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:29.731145 (+ 2184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
W20260927 03:23:29.731287 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:29.725107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50147 (local address 127.0.98.62:40217)
0927 03:23:29.725167 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:29.725169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:29.725298 (+ 129us) server_negotiation.cc:407] Connection header received
0927 03:23:29.725755 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:29.725756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:29.725765 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:29.725817 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:29.726690 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.727254 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.728222 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.728411 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.728468 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.731229 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50147: BlockingRecv error: recv got EOF from 127.0.98.3:50147 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 03:23:29.731508 9526 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40217 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40217: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260927 03:23:29.732200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 9396
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:29.736325 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:29.731724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:54693)
0927 03:23:29.731769 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:29.731779 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:29.731817 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:29.732081 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:29.732082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:29.732209 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:29.732263 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.732265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.733069 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.733070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.733764 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.733766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.734178 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.734178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.734221 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.734571 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:29.734576 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:29.736261 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:23:29.736398 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:29.731857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54693 (local address 127.0.98.62:40217)
0927 03:23:29.731956 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:29.731957 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:29.731964 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:23:29.731983 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:29.731984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:29.731991 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:29.732040 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:29.732370 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.732994 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.733830 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.734108 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.734169 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.736346 (+ 2177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54693: BlockingRecv error: recv got EOF from 127.0.98.3:54693 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:29.739709 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:29.732585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.0.98.62:40217)
0927 03:23:29.732682 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:29.732683 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:29.732770 (+ 87us) server_negotiation.cc:407] Connection header received
0927 03:23:29.732843 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:29.732843 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:29.732853 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:29.732905 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:29.733560 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.734204 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.735234 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.735444 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.735510 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.736726 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:29.736731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:29.736731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:29.736743 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:29.738195 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:29.738507 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:29.738510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:29.738510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:29.738619 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:29.739030 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:29.739031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:29.739031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:29.739238 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:29.739363 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:29.739484 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 03:23:29.739616 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":51,"threads_started":1}
Sep 27 03:23:29 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:29.740494 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:29.736582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:44979)
0927 03:23:29.736609 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:29.736619 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:29.736653 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:29.736762 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:29.736763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:29.736875 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:29.736900 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.736901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.737672 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.737673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.738165 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:29.738167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.738513 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.738514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:29.738559 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.738887 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:29.738891 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:29.740434 (+ 1543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":14}
W20260927 03:23:29.740640 9526 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:23:29.740679 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:29.736637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44979 (local address 127.0.98.62:40217)
0927 03:23:29.736675 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:29.736676 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:29.736683 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:23:29.736701 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:29.736702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:29.736710 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:29.736754 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:29.736970 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.737605 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.738237 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:29.738442 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:29.738498 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:29.740489 (+ 1991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44979: BlockingRecv error: recv got EOF from 127.0.98.3:44979 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 27 03:23:30 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:30 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:30.493040 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:30.488668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:36137)
0927 03:23:30.488797 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:30.488809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:30.488880 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:30.488999 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:30.489000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:30.489147 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:30.489194 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:30.489196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:30.489845 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:30.489845 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:30.490382 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:30.490384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:30.490704 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:30.490705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:30.490745 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:30.491183 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:30.491188 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:30.492971 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
W20260927 03:23:30.493124 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:30.488761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36137 (local address 127.0.98.62:40217)
0927 03:23:30.488883 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:30.488884 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:30.488910 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:23:30.488929 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:30.488930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:30.488939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:30.488984 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:30.489284 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:30.489780 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:30.490466 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:30.490651 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:30.490700 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:30.493061 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36137: BlockingRecv error: recv got EOF from 127.0.98.1:36137 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
Sep 27 03:23:30 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:30 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:30.628074 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:30.623335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:39721)
0927 03:23:30.623470 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:30.623479 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:30.623540 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:30.623662 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:30.623662 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:30.623820 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:30.623870 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:30.623871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:30.624530 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:30.624532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:30.625165 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:30.625168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:30.625529 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:30.625530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:30.625583 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:30.626097 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:30.626102 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:30.627981 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":59,"threads_started":1}
W20260927 03:23:30.628134 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:30.623373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39721 (local address 127.0.98.62:40217)
0927 03:23:30.623427 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:30.623429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:30.623525 (+ 96us) server_negotiation.cc:407] Connection header received
0927 03:23:30.623582 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:30.623583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:30.623593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:30.623648 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:30.623950 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:30.624428 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:30.625243 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:30.625435 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:30.625488 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:30.628082 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39721: BlockingRecv error: recv got EOF from 127.0.98.2:39721 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 27 03:23:30 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:30 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:30.745667 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:30.741080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:53421)
0927 03:23:30.741205 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:30.741215 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:30.741269 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:30.741389 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:30.741390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:30.741571 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:30.741617 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:30.741619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:30.742260 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:30.742261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:30.742885 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:30.742888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:30.743215 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:30.743216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:30.743271 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:30.743704 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:30.743708 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:30.745598 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
W20260927 03:23:30.745757 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:30.741221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53421 (local address 127.0.98.62:40217)
0927 03:23:30.741282 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:30.741284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:30.741292 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:30.741317 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:30.741318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:30.741331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:30.741375 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:30.741691 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:30.742186 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:30.742972 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:30.743158 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:30.743210 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:30.745693 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53421: BlockingRecv error: recv got EOF from 127.0.98.3:53421 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 03:23:31 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:31 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:31.498106 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:31.493583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:58243)
0927 03:23:31.493704 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:31.493720 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:31.493774 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:31.493949 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:31.493950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:31.494111 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:31.494159 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:31.494161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:31.494875 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:31.494876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:31.495415 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:31.495416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:31.495736 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:31.495736 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:31.495778 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:31.496238 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:31.496242 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:31.498029 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
W20260927 03:23:31.498226 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:31.493651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58243 (local address 127.0.98.62:40217)
0927 03:23:31.493767 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:31.493769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:31.493782 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:23:31.493811 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:31.493811 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:31.493823 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:31.493893 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:31.494243 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:31.494752 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:31.495487 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:31.495681 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:31.495731 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:31.498132 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58243: BlockingRecv error: recv got EOF from 127.0.98.1:58243 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
Sep 27 03:23:31 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:31 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:31.633545 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:31.628663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:43481)
0927 03:23:31.628830 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:31.628842 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:31.628910 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:31.629043 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:31.629044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:31.629211 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:31.629267 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:31.629269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:31.629926 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:31.629927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:31.630479 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:31.630482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:31.631044 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:31.631045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:31.631114 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:31.631624 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:31.631629 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:31.633459 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":142,"threads_started":1}
W20260927 03:23:31.633625 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:31.628857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43481 (local address 127.0.98.62:40217)
0927 03:23:31.628926 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:31.628928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:31.628938 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:31.628966 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:31.628967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:31.628980 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:31.629028 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:31.629344 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:31.629850 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:31.630567 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:31.630949 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:31.631029 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:31.633572 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43481: BlockingRecv error: recv got EOF from 127.0.98.2:43481 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 03:23:31 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:31 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:31.750762 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:31.746135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:33803)
0927 03:23:31.746292 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:31.746304 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:31.746366 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:31.746497 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:31.746498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:31.746653 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:31.746705 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:31.746707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:31.747397 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:31.747398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:31.747942 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:31.747944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:31.748283 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:31.748284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:31.748326 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:31.748865 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:31.748870 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:31.750685 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":57,"threads_started":1}
W20260927 03:23:31.750861 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:31.746272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33803 (local address 127.0.98.62:40217)
0927 03:23:31.746346 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:31.746348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:31.746358 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:31.746422 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:31.746422 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:31.746434 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:31.746483 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:31.746804 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:31.747317 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:31.748032 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:31.748220 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:31.748274 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:31.750778 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33803: BlockingRecv error: recv got EOF from 127.0.98.3:33803 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 03:23:32 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:32 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:32.503675 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:32.498586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:58169)
0927 03:23:32.498724 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:32.498734 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:32.498796 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:32.499024 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:32.499025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:32.499180 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:32.499226 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:32.499227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:32.500033 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:32.500034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:32.500618 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:32.500633 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:32.501006 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:32.501007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:32.501060 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:32.501553 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:32.501558 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:32.503609 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
W20260927 03:23:32.503815 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:32.498650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58169 (local address 127.0.98.62:40217)
0927 03:23:32.498783 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:32.498785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:32.498795 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:32.498903 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:32.498904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:32.498918 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:32.498991 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:32.499316 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:32.499939 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:32.500717 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:32.500918 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:32.500974 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:32.503728 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58169: BlockingRecv error: recv got EOF from 127.0.98.1:58169 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
Sep 27 03:23:32 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:32 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:32.638967 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:32.634073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:51163)
0927 03:23:32.634228 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:32.634239 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:32.634290 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:32.634475 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:32.634476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:32.634654 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:32.634704 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:32.634706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:32.635410 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:32.635411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:32.635963 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:32.635965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:32.636321 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:32.636322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:32.636375 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:32.636894 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:32.636899 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:32.638884 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
W20260927 03:23:32.639091 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:32.634189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51163 (local address 127.0.98.62:40217)
0927 03:23:32.634273 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:32.634275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:32.634286 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:23:32.634354 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:32.634355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:32.634366 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:32.634425 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:32.634801 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:32.635333 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:32.636041 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:32.636240 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:32.636292 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:32.638998 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51163: BlockingRecv error: recv got EOF from 127.0.98.2:51163 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 03:23:32 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:32 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:32.756247 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:32.751362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:38853)
0927 03:23:32.751515 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:32.751527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:32.751660 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:32.751790 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:32.751790 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:32.751947 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:32.751994 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:32.751995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:32.752690 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:32.752691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:32.753226 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:32.753229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:32.753574 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:32.753575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:32.753621 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:32.754174 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:32.754178 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:32.756159 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
W20260927 03:23:32.756311 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:32.751482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38853 (local address 127.0.98.62:40217)
0927 03:23:32.751528 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:32.751530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:32.751639 (+ 109us) server_negotiation.cc:407] Connection header received
0927 03:23:32.751721 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:32.751723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:32.751736 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:32.751774 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:32.752046 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:32.752553 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:32.753313 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:32.753503 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:32.753559 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:32.756260 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38853: BlockingRecv error: recv got EOF from 127.0.98.3:38853 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 27 03:23:33 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:33 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:33.508433 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:33.504160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:57583)
0927 03:23:33.504312 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:33.504324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:33.504393 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:33.504509 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:33.504510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:33.504666 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:33.504713 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:33.504715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:33.505350 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:33.505351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:33.505881 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:33.505883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:33.506175 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:33.506176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:33.506217 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:33.506653 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:33.506658 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:33.508355 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
W20260927 03:23:33.508507 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:33.504226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57583 (local address 127.0.98.62:40217)
0927 03:23:33.504327 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:33.504329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:33.504389 (+ 60us) server_negotiation.cc:407] Connection header received
0927 03:23:33.504437 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:33.504438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:33.504449 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:33.504505 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:33.504768 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:33.505285 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:33.505939 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:33.506123 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:33.506164 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:33.508444 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57583: BlockingRecv error: recv got EOF from 127.0.98.1:57583 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":44,"threads_started":1}
Sep 27 03:23:33 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:33 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:33.643994 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:33.639463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:46779)
0927 03:23:33.639621 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:33.639632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:33.639711 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:33.639942 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:33.639943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:33.640149 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:33.640208 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:33.640210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:33.640930 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:33.640930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:33.641458 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:33.641460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:33.641777 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:33.641778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:33.641816 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:33.642235 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:33.642239 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:33.643936 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":61,"threads_started":1}
W20260927 03:23:33.644089 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:33.639533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46779 (local address 127.0.98.62:40217)
0927 03:23:33.639583 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:33.639585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:33.639694 (+ 109us) server_negotiation.cc:407] Connection header received
0927 03:23:33.639773 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:33.639774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:33.639787 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:33.639835 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:33.640282 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:33.640855 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:33.641527 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:33.641721 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:33.641772 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:33.644031 (+ 2259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46779: BlockingRecv error: recv got EOF from 127.0.98.2:46779 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 27 03:23:33 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:33 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:33.761371 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:33.756771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:36095)
0927 03:23:33.756908 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:33.756920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:33.757000 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:33.757180 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:33.757181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:33.757353 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:33.757398 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:33.757400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:33.758081 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:33.758082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:33.758651 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:33.758654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:33.759040 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:33.759041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:33.759096 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:33.759605 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:33.759611 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:33.761310 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
W20260927 03:23:33.761463 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:33.756862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36095 (local address 127.0.98.62:40217)
0927 03:23:33.756920 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:33.756922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:33.756991 (+ 69us) server_negotiation.cc:407] Connection header received
0927 03:23:33.757055 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:33.757057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:33.757079 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:33.757160 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:33.757504 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:33.758004 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:33.758740 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:33.758977 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:33.759041 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:33.761401 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36095: BlockingRecv error: recv got EOF from 127.0.98.3:36095 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 03:23:34 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:34 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:34.516899 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:34.508870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:36847)
0927 03:23:34.508994 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:34.509004 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:34.509062 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:34.509177 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:34.509178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:34.509342 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:34.509389 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:34.509390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:34.510021 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:34.510022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:34.510546 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:34.510548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:34.510890 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:34.510890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:34.510930 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:34.514978 (+ 4048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:34.514984 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:34.516822 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
W20260927 03:23:34.516974 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:34.508921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36847 (local address 127.0.98.62:40217)
0927 03:23:34.509016 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:34.509018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:34.509047 (+ 29us) server_negotiation.cc:407] Connection header received
0927 03:23:34.509102 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:34.509102 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:34.509113 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:34.509166 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:34.509442 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:34.509954 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:34.510617 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:34.510839 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:34.510890 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:34.516912 (+ 6022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36847: BlockingRecv error: recv got EOF from 127.0.98.1:36847 (error 108)
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":43,"threads_started":1}
Sep 27 03:23:34 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:34 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:34.649497 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:34.644591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:37731)
0927 03:23:34.644753 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:34.644764 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:34.644825 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:34.644987 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:34.644987 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:34.645193 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:34.645244 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:34.645246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:34.646148 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:34.646149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:34.646745 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:34.646747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:34.647143 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:34.647144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:34.647186 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:34.647611 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:34.647615 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:34.649421 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":66,"threads_started":1}
W20260927 03:23:34.649565 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:34.644762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37731 (local address 127.0.98.62:40217)
0927 03:23:34.644860 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:34.644862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:34.644872 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:34.644900 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:34.644900 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:34.644915 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:34.644973 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:34.645318 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:34.646084 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:34.646845 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:34.647058 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:34.647121 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:34.649514 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37731: BlockingRecv error: recv got EOF from 127.0.98.2:37731 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 03:23:34 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:34 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:34.766498 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:34.761859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:50089)
0927 03:23:34.761991 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:34.762004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:34.762081 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:34.762249 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:34.762250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:34.762410 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:34.762456 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:34.762457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:34.763139 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:34.763140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:34.763666 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:34.763667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:34.763977 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:34.763977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:34.764029 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:34.764469 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:34.764474 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:34.766439 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
W20260927 03:23:34.766597 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:34.761957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50089 (local address 127.0.98.62:40217)
0927 03:23:34.762033 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:34.762035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:34.762075 (+ 40us) server_negotiation.cc:407] Connection header received
0927 03:23:34.762178 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:34.762179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:34.762192 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:34.762230 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:34.762520 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:34.763072 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:34.763736 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:34.763920 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:34.763973 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:34.766533 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50089: BlockingRecv error: recv got EOF from 127.0.98.3:50089 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 03:23:35 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:35 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:35.521839 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:35.517427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:32953)
0927 03:23:35.517594 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:35.517605 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:35.517676 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:35.517810 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:35.517811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:35.517963 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:35.518009 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:35.518011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:35.518630 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:35.518631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:35.519213 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:35.519215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:35.519540 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:35.519540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:35.519575 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:35.520068 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:35.520074 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:35.521773 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":74,"threads_started":1}
W20260927 03:23:35.521961 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:35.517519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32953 (local address 127.0.98.62:40217)
0927 03:23:35.517625 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:35.517626 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:35.517658 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:23:35.517727 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:35.517728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:35.517738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:35.517795 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:35.518061 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:35.518565 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:35.519274 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:35.519483 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:35.519533 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:35.521864 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32953: BlockingRecv error: recv got EOF from 127.0.98.1:32953 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":48,"threads_started":1}
Sep 27 03:23:35 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:35 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:35.654492 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:35.649985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:56203)
0927 03:23:35.650092 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:35.650103 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:35.650175 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:35.650302 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:35.650302 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:35.650457 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:35.650507 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:35.650508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:35.651170 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:35.651171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:35.651708 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:35.651710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:35.652146 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:35.652148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:35.652194 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:35.652666 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:35.652671 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:35.654427 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
W20260927 03:23:35.654568 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:35.650054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56203 (local address 127.0.98.62:40217)
0927 03:23:35.650096 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:35.650098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:35.650158 (+ 60us) server_negotiation.cc:407] Connection header received
0927 03:23:35.650228 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:35.650228 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:35.650241 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:35.650291 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:35.650572 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:35.651113 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:35.651814 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:35.652073 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:35.652143 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:35.654521 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56203: BlockingRecv error: recv got EOF from 127.0.98.2:56203 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 27 03:23:35 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:35 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:35.772197 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:35.767089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:34407)
0927 03:23:35.767247 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:35.767261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:35.767416 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:35.767583 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:35.767584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:35.767751 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:35.767809 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:35.767811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:35.768527 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:35.768528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:35.769088 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:35.769090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:35.769501 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:35.769502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:35.769553 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:35.770053 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:35.770059 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:35.772127 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":72,"threads_started":1}
W20260927 03:23:35.772277 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:35.767201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34407 (local address 127.0.98.62:40217)
0927 03:23:35.767283 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:35.767285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:35.767421 (+ 136us) server_negotiation.cc:407] Connection header received
0927 03:23:35.767469 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:35.767470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:35.767485 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:35.767554 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:35.767881 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:35.768388 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:35.769159 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:35.769344 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:35.769398 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:35.772218 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34407: BlockingRecv error: recv got EOF from 127.0.98.3:34407 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 03:23:36 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:36 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:36.526748 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:36.522291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:40419)
0927 03:23:36.522415 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:36.522426 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:36.522480 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:36.522602 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:36.522603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:36.522757 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:36.522806 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:36.522808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:36.523479 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:36.523480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:36.524005 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:36.524007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:36.524377 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:36.524378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:36.524417 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:36.524857 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:36.524861 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:36.526661 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
W20260927 03:23:36.526870 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:36.522357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40419 (local address 127.0.98.62:40217)
0927 03:23:36.522482 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:36.522483 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:36.522515 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:23:36.522535 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:36.522535 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:36.522546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:36.522589 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:36.522897 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:36.523416 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:36.524096 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:36.524309 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:36.524379 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:36.526773 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40419: BlockingRecv error: recv got EOF from 127.0.98.1:40419 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
Sep 27 03:23:36 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:36 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:36.659608 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:36.654995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:50881)
0927 03:23:36.655158 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:36.655169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:36.655227 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:36.655349 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:36.655349 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:36.655504 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:36.655553 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:36.655554 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:36.656153 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:36.656154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:36.656744 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:36.656746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:36.657084 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:36.657084 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:36.657122 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:36.657563 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:36.657568 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:36.659537 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
W20260927 03:23:36.659693 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:36.655056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50881 (local address 127.0.98.62:40217)
0927 03:23:36.655150 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:36.655152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:36.655223 (+ 71us) server_negotiation.cc:407] Connection header received
0927 03:23:36.655284 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:36.655285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:36.655297 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:36.655337 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:36.655604 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:36.656095 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:36.656825 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:36.657028 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:36.657082 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:36.659632 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50881: BlockingRecv error: recv got EOF from 127.0.98.2:50881 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 03:23:36 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:36 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:36.777285 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:36.772679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:56269)
0927 03:23:36.772837 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:36.772849 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:36.772919 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:36.773048 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:36.773049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:36.773250 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:36.773307 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:36.773310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:36.774045 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:36.774046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:36.774618 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:36.774620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:36.775038 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:36.775039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:36.775098 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:36.775556 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:36.775561 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:36.777226 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":68,"threads_started":1}
W20260927 03:23:36.777372 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:36.772747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56269 (local address 127.0.98.62:40217)
0927 03:23:36.772823 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:36.772825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:36.772913 (+ 88us) server_negotiation.cc:407] Connection header received
0927 03:23:36.772964 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:36.772964 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:36.772977 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:36.773031 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:36.773370 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:36.773973 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:36.774700 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:36.774964 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:36.775019 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:36.777313 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56269: BlockingRecv error: recv got EOF from 127.0.98.3:56269 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 03:23:37 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:37 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:37.531788 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:37.527304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:58529)
0927 03:23:37.527483 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:37.527496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:37.527564 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:37.527727 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:37.527728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:37.527969 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:37.528025 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:37.528028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:37.528744 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:37.528745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:37.529288 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:37.529290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:37.529619 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:37.529620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:37.529668 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:37.530103 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:37.530109 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:37.531721 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":89,"threads_started":1}
W20260927 03:23:37.531899 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:37.527442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58529 (local address 127.0.98.62:40217)
0927 03:23:37.527580 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:37.527582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:37.527591 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:23:37.527614 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:37.527615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:37.527630 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:37.527710 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:37.528119 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:37.528670 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:37.529359 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:37.529563 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:37.529614 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:37.531821 (+ 2207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58529: BlockingRecv error: recv got EOF from 127.0.98.1:58529 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
Sep 27 03:23:37 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:37 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:37.665714 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:37.660039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:55561)
0927 03:23:37.660185 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:37.660198 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:37.660262 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:37.660382 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:37.660383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:37.660580 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:37.660634 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:37.660637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:37.661761 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:37.661763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:37.662593 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:37.662595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:37.662995 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:37.662996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:37.663049 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:37.663528 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:37.663535 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:37.665634 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
W20260927 03:23:37.665778 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:37.660127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55561 (local address 127.0.98.62:40217)
0927 03:23:37.660208 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:37.660210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:37.660294 (+ 84us) server_negotiation.cc:407] Connection header received
0927 03:23:37.660315 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:37.660316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:37.660327 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:37.660368 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:37.660697 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:37.661685 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:37.662672 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:37.662938 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:37.662995 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:37.665729 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55561: BlockingRecv error: recv got EOF from 127.0.98.2:55561 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 03:23:37 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:37 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:37.782361 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:37.777796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:42287)
0927 03:23:37.777926 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:37.777937 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:37.778002 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:37.778210 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:37.778211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:37.778390 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:37.778446 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:37.778448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:37.779133 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:37.779134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:37.779673 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:37.779674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:37.780017 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:37.780018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:37.780086 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:37.780582 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:37.780587 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:37.782300 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
W20260927 03:23:37.782518 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:37.777880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42287 (local address 127.0.98.62:40217)
0927 03:23:37.777928 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:37.777930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:37.777995 (+ 65us) server_negotiation.cc:407] Connection header received
0927 03:23:37.778062 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:37.778063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:37.778077 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:37.778137 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:37.778533 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:37.779075 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:37.779759 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:37.779941 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:37.779992 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:37.782450 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42287: BlockingRecv error: recv got EOF from 127.0.98.3:42287 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 03:23:38 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:38 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:38.536651 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:38.532255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:37685)
0927 03:23:38.532370 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:38.532382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:38.532445 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:38.532554 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:38.532554 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:38.532716 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:38.532766 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:38.532768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:38.533438 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:38.533438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:38.533987 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:38.533989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:38.534330 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:38.534331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:38.534367 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:38.534775 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:38.534780 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:38.536589 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
W20260927 03:23:38.536763 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:38.532335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37685 (local address 127.0.98.62:40217)
0927 03:23:38.532451 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:38.532453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:38.532463 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:38.532486 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:38.532486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:38.532496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:38.532549 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:38.532859 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:38.533364 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:38.534070 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:38.534271 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:38.534325 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:38.536692 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37685: BlockingRecv error: recv got EOF from 127.0.98.1:37685 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
Sep 27 03:23:38 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:38 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:38.670933 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:38.666171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:59419)
0927 03:23:38.666338 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:38.666352 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:38.666422 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:38.666541 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:38.666542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:38.666700 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:38.666750 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:38.666751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:38.667491 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:38.667492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:38.668062 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:38.668064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:38.668391 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:38.668391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:38.668426 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:38.668891 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:38.668895 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:38.670873 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
W20260927 03:23:38.671018 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:38.666304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59419 (local address 127.0.98.62:40217)
0927 03:23:38.666365 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:38.666367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:38.666406 (+ 39us) server_negotiation.cc:407] Connection header received
0927 03:23:38.666477 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:38.666478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:38.666489 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:38.666528 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:38.666862 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:38.667418 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:38.668140 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:38.668335 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:38.668388 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:38.670954 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59419: BlockingRecv error: recv got EOF from 127.0.98.2:59419 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 03:23:38 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:38 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:38.787504 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:38.782867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:34967)
0927 03:23:38.783006 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:38.783017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:38.783077 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:38.783328 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:38.783330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:38.783489 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:38.783535 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:38.783537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:38.784203 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:38.784205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:38.784737 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:38.784739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:38.785058 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:38.785059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:38.785096 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:38.785554 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:38.785558 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:38.787428 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
W20260927 03:23:38.787593 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:38.782927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34967 (local address 127.0.98.62:40217)
0927 03:23:38.783002 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:38.783004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:38.783071 (+ 67us) server_negotiation.cc:407] Connection header received
0927 03:23:38.783123 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:38.783124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:38.783135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:38.783190 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:38.783605 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:38.784123 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:38.784813 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:38.784995 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:38.785047 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:38.787539 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34967: BlockingRecv error: recv got EOF from 127.0.98.3:34967 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 03:23:39 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:39 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:39.545516 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:39.537172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:40193)
0927 03:23:39.537320 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:39.537331 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:39.537387 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:39.537588 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:39.537589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:39.537764 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:39.537814 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:39.537816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:39.538520 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:39.538521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:39.539135 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:39.539137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:39.539534 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:39.539535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:39.539579 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:39.543614 (+ 4035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:39.543621 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:39.545444 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
W20260927 03:23:39.545686 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:39.537274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40193 (local address 127.0.98.62:40217)
0927 03:23:39.537379 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:39.537381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:39.537390 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:23:39.537435 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:39.537435 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:39.537446 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:39.537537 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:39.537901 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:39.538429 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:39.539226 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:39.539436 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:39.539528 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:39.545612 (+ 6084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40193: BlockingRecv error: recv got EOF from 127.0.98.1:40193 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":44,"threads_started":1}
Sep 27 03:23:39 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:39 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:39.676396 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:39.671410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:50765)
0927 03:23:39.671538 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:39.671549 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:39.671612 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:39.671753 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:39.671754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:39.671918 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:39.672038 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:39.672040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:39.672956 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:39.672957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:39.673570 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:39.673573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:39.673904 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:39.673905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:39.673952 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:39.674380 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:39.674385 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:39.676336 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
W20260927 03:23:39.676517 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:39.671484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50765 (local address 127.0.98.62:40217)
0927 03:23:39.671548 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:39.671549 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:39.671597 (+ 48us) server_negotiation.cc:407] Connection header received
0927 03:23:39.671664 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:39.671665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:39.671675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:39.671737 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:39.672185 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:39.672880 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:39.673642 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:39.673843 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:39.673897 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:39.676447 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50765: BlockingRecv error: recv got EOF from 127.0.98.2:50765 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 27 03:23:39 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:39 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:39.792225 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:39.788025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:46587)
0927 03:23:39.788181 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:39.788197 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:39.788257 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:39.788406 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:39.788406 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:39.788575 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:39.788626 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:39.788628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:39.789270 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:39.789271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:39.789790 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:39.789791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:39.790090 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:39.790090 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:39.790131 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:39.790531 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:39.790534 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:39.792166 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
W20260927 03:23:39.792311 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:39.788155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46587 (local address 127.0.98.62:40217)
0927 03:23:39.788197 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:39.788198 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:39.788243 (+ 45us) server_negotiation.cc:407] Connection header received
0927 03:23:39.788316 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:39.788316 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:39.788328 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:39.788367 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:39.788689 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:39.789209 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:39.789859 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:39.790039 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:39.790110 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:39.792250 (+ 2140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46587: BlockingRecv error: recv got EOF from 127.0.98.3:46587 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 03:23:40 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:40 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:40.550352 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:40.546014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:51221)
0927 03:23:40.546136 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:40.546149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:40.546211 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:40.546344 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:40.546345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:40.546497 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:40.546587 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:40.546589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:40.547248 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:40.547249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:40.547786 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:40.547787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:40.548141 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:40.548142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:40.548176 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:40.548627 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:40.548631 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:40.550290 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
W20260927 03:23:40.550465 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:40.546087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51221 (local address 127.0.98.62:40217)
0927 03:23:40.546203 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:40.546205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:40.546215 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:40.546255 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:40.546256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:40.546269 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:40.546342 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:40.546669 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:40.547193 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:40.547878 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:40.548077 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:40.548140 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:40.550389 (+ 2249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51221: BlockingRecv error: recv got EOF from 127.0.98.1:51221 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
Sep 27 03:23:40 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:40 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:40.680943 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:40.676829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:35557)
0927 03:23:40.676949 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:40.676959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:40.677014 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:40.677140 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:40.677141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:40.677293 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:40.677342 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:40.677343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:40.677935 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:40.677936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:40.678461 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:40.678462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:40.678754 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:40.678754 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:40.678793 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:40.679255 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:40.679259 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:40.680886 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
W20260927 03:23:40.681043 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:40.676921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35557 (local address 127.0.98.62:40217)
0927 03:23:40.676976 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:40.676978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:40.677014 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:23:40.677061 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:40.677061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:40.677072 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:40.677129 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:40.677391 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:40.677868 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:40.678524 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:40.678703 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:40.678748 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:40.680976 (+ 2228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35557: BlockingRecv error: recv got EOF from 127.0.98.2:35557 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 03:23:40 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:40 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:40.796990 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:40.792694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:45249)
0927 03:23:40.792828 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:40.792839 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:40.792900 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:40.793012 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:40.793012 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:40.793163 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:40.793210 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:40.793212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:40.793859 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:40.793861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:40.794465 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:40.794467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:40.794846 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:40.794847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:40.794909 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:40.795351 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:40.795356 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:40.796930 (+ 1574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
W20260927 03:23:40.797083 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:40.792791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45249 (local address 127.0.98.62:40217)
0927 03:23:40.792846 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:40.792848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:40.792892 (+ 44us) server_negotiation.cc:407] Connection header received
0927 03:23:40.792940 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:40.792941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:40.792953 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:40.792995 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:40.793257 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:40.793761 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:40.794561 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:40.794743 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:40.794792 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:40.797020 (+ 2228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45249: BlockingRecv error: recv got EOF from 127.0.98.3:45249 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 03:23:41 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:41 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:41.555318 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:41.550911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:58669)
0927 03:23:41.551042 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:41.551053 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:41.551104 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:41.551276 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:41.551277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:41.551483 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:41.551544 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:41.551547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:41.552187 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:41.552188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:41.552707 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:41.552708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:41.553024 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:41.553025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:41.553059 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:41.553450 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:41.553454 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:41.555249 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
W20260927 03:23:41.555428 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:41.551028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58669 (local address 127.0.98.62:40217)
0927 03:23:41.551145 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:41.551147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:41.551155 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:41.551177 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:41.551178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:41.551189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:41.551260 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:41.551620 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:41.552129 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:41.552773 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:41.552972 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:41.553022 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:41.555343 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58669: BlockingRecv error: recv got EOF from 127.0.98.1:58669 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
Sep 27 03:23:41 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:41 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:41.686110 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:41.681397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:36027)
0927 03:23:41.681554 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:41.681564 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:41.681621 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:41.681731 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:41.681731 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:41.681889 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:41.681936 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:41.681937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:41.682607 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:41.682608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:41.683262 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:41.683264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:41.683612 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:41.683613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:41.683649 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:41.684190 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:41.684194 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:41.686042 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
W20260927 03:23:41.686203 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:41.681467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36027 (local address 127.0.98.62:40217)
0927 03:23:41.681528 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:41.681530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:41.681616 (+ 86us) server_negotiation.cc:407] Connection header received
0927 03:23:41.681664 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:41.681665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:41.681676 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:41.681719 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:41.682029 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:41.682528 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:41.683355 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:41.683553 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:41.683615 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:41.686143 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36027: BlockingRecv error: recv got EOF from 127.0.98.2:36027 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 03:23:41 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:41 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:41.802206 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:41.797506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:57533)
0927 03:23:41.797654 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:41.797665 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:41.797735 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:41.797856 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:41.797857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:41.798011 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:41.798060 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:41.798061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:41.798794 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:41.798796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:41.799374 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:41.799376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:41.799713 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:41.799714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:41.799822 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:41.800253 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:41.800258 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:41.802146 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":61,"threads_started":1}
W20260927 03:23:41.802304 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:41.797627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57533 (local address 127.0.98.62:40217)
0927 03:23:41.797681 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:41.797683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:41.797718 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:23:41.797784 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:41.797785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:41.797797 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:41.797840 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:41.798109 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:41.798653 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:41.799445 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:41.799630 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:41.799682 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:41.802237 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57533: BlockingRecv error: recv got EOF from 127.0.98.3:57533 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 03:23:42 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:42 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:42.560459 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:42.555780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:35811)
0927 03:23:42.555928 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:42.555938 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:42.555991 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:42.556116 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:42.556116 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:42.556266 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:42.556315 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:42.556316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:42.556968 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:42.556969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:42.557520 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:42.557522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:42.558015 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:42.558016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:42.558061 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:42.558565 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:42.558570 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:42.560398 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":58,"threads_started":1}
W20260927 03:23:42.560562 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:42.555832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35811 (local address 127.0.98.62:40217)
0927 03:23:42.555946 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:42.555947 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:42.555979 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:23:42.556044 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:42.556044 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:42.556055 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:42.556102 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:42.556368 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:42.556895 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:42.557653 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:42.557932 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:42.558000 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:42.560495 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35811: BlockingRecv error: recv got EOF from 127.0.98.1:35811 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
Sep 27 03:23:42 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:42 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:42.691094 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:42.686637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:44941)
0927 03:23:42.686882 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:42.686893 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:42.686959 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:42.687085 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:42.687085 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:42.687240 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:42.687289 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:42.687290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:42.687926 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:42.687927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:42.688474 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:42.688476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:42.688799 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:42.688800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:42.688841 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:42.689269 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:42.689273 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:42.691036 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":159,"threads_started":1}
W20260927 03:23:42.691193 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:42.686792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44941 (local address 127.0.98.62:40217)
0927 03:23:42.686890 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:42.686892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:42.686953 (+ 61us) server_negotiation.cc:407] Connection header received
0927 03:23:42.687019 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:42.687019 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:42.687030 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:42.687072 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:42.687338 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:42.687862 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:42.688545 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:42.688735 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:42.688786 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:42.691134 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44941: BlockingRecv error: recv got EOF from 127.0.98.2:44941 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 03:23:42 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:42 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:42.807281 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:42.802695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:51935)
0927 03:23:42.802840 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:42.802851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:42.802930 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:42.803078 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:42.803078 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:42.803310 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:42.803360 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:42.803363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:42.803994 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:42.803995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:42.804530 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:42.804531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:42.804872 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:42.804873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:42.804922 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:42.805379 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:42.805384 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:42.807203 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
W20260927 03:23:42.807343 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:42.802754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51935 (local address 127.0.98.62:40217)
0927 03:23:42.802804 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:42.802806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:42.802928 (+ 122us) server_negotiation.cc:407] Connection header received
0927 03:23:42.802996 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:42.802997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:42.803011 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:42.803059 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:42.803430 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:42.803925 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:42.804596 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:42.804789 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:42.804841 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:42.807292 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51935: BlockingRecv error: recv got EOF from 127.0.98.3:51935 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 27 03:23:43 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:43 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:43.565840 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:43.560995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:38557)
0927 03:23:43.561144 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:43.561157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:43.561238 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:43.561413 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:43.561414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:43.561632 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:43.561681 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:43.561683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:43.562339 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:43.562341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:43.563072 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:43.563075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:43.563504 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:43.563505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:43.563576 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:43.564042 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:43.564047 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:43.565780 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
W20260927 03:23:43.565948 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:43.561117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38557 (local address 127.0.98.62:40217)
0927 03:23:43.561261 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:43.561264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:43.561275 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:23:43.561300 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:43.561301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:43.561310 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:43.561369 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:43.561767 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:43.562258 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:43.563153 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:43.563435 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:43.563498 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:43.565881 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38557: BlockingRecv error: recv got EOF from 127.0.98.1:38557 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
Sep 27 03:23:43 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:43 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:43.696141 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:43.691589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:44955)
0927 03:23:43.691734 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:43.691746 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:43.691813 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:43.691933 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:43.691934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:43.692091 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:43.692139 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:43.692140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:43.692779 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:43.692781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:43.693511 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:43.693515 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:43.693840 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:43.693840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:43.693876 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:43.694305 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:43.694309 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:43.696073 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
W20260927 03:23:43.696220 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:43.691675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44955 (local address 127.0.98.62:40217)
0927 03:23:43.691750 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:43.691752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:43.691806 (+ 54us) server_negotiation.cc:407] Connection header received
0927 03:23:43.691871 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:43.691871 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:43.691882 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:43.691919 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:43.692191 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:43.692682 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:43.693582 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:43.693784 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:43.693836 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:43.696160 (+ 2324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44955: BlockingRecv error: recv got EOF from 127.0.98.2:44955 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 03:23:43 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:43 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:43.812479 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:43.807847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.3:35069)
0927 03:23:43.808021 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:43.808033 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:43.808094 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:43.808263 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:43.808264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:43.808423 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:43.808468 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:43.808470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:43.809150 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:43.809152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:43.809671 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:43.809673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:43.810062 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:43.810063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:43.810118 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:43.810586 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:43.810590 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:43.812409 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":114,"threads_started":1}
W20260927 03:23:43.812549 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:43.808004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35069 (local address 127.0.98.62:40217)
0927 03:23:43.808069 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:43.808071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:43.808097 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:23:43.808153 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:43.808154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:43.808170 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:43.808215 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:43.808553 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:43.809079 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:43.809781 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:43.809993 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:43.810045 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:43.812490 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35069: BlockingRecv error: recv got EOF from 127.0.98.3:35069 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 03:23:44 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:44 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:44.576010 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:44.566420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.1:54075)
0927 03:23:44.566554 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:44.566568 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:44.566640 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:44.566855 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:44.566856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:44.567082 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:44.567140 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:44.567142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:44.567887 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:44.567888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:44.568434 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:44.568436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:44.568822 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:44.568822 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:44.568864 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:44.573917 (+ 5053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:44.573923 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:44.575937 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
W20260927 03:23:44.576155 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:44.566501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54075 (local address 127.0.98.62:40217)
0927 03:23:44.566718 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:44.566720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:44.566730 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:44.566758 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:44.566758 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:44.566768 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:44.566845 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:44.567207 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:44.567787 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:44.568558 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:44.568758 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:44.568813 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:44.576075 (+ 7262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54075: BlockingRecv error: recv got EOF from 127.0.98.1:54075 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":120,"threads_started":1}
Sep 27 03:23:44 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:23:44 dist-test-slave-gw2v krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:23:44.701404 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:44.696695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.98.2:46429)
0927 03:23:44.696811 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:44.696821 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:44.696868 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:44.697006 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:44.697007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:44.697191 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:44.697240 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:44.697241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:44.697975 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:44.697976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:44.698517 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:44.698519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:44.698968 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:44.698969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:44.699012 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:44.699521 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:44.699525 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:44.701312 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40217: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
W20260927 03:23:44.701455 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:44.696773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46429 (local address 127.0.98.62:40217)
0927 03:23:44.696848 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:44.696850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:44.696860 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:23:44.696926 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:44.696926 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:44.696936 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:44.696996 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:44.697341 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:44.697868 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:44.698607 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:44.698805 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:44.698968 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:44.701404 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46429: BlockingRecv error: recv got EOF from 127.0.98.2:46429 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 03:23:44.733295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 9112
I20260927 03:23:44.737219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 9264
I20260927 03:23:44.741103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 9396
I20260927 03:23:44.744827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 9042
2026-09-27T03:23:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17295 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T03:23:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:44Z Disabled control of system clock
I20260927 03:23:44.771059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34045
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 03:23:44.849761 9593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:44.849974 9593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:44.849994 9593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:44.851166 9593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:44.851205 9593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:44.851217 9593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:44.851228 9593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:44.851239 9593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:44.852555 9593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34045
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:44.852747 9593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:44.852994 9593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:44.857793 9599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:44.857780 9598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:44.857810 9601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:44.857990 9593 server_base.cc:1034] running on GCE node
I20260927 03:23:44.858274 9593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:44.858569 9593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:44.859727 9593 hybrid_clock.cc:648] HybridClock initialized: now 1790479424859713 us; error 30 us; skew 500 ppm
I20260927 03:23:44.861064 9593 webserver.cc:460] Webserver started at http://127.0.98.62:35381/ using document root <none> and password file <none>
I20260927 03:23:44.861337 9593 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:44.861389 9593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:44.861513 9593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:44.862565 9593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "286a40b4f3384139918cbc33892d556e"
format_stamp: "Formatted at 2026-09-27 03:23:44 on dist-test-slave-gw2v"
server_key: "eff1c9bb2a20d48ef3b3e439702e1717"
server_key_iv: "cc75cb96333c61d15cb5b118798767f1"
server_key_version: "encryptionkey@0"
I20260927 03:23:44.862921 9593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "286a40b4f3384139918cbc33892d556e"
format_stamp: "Formatted at 2026-09-27 03:23:44 on dist-test-slave-gw2v"
server_key: "eff1c9bb2a20d48ef3b3e439702e1717"
server_key_iv: "cc75cb96333c61d15cb5b118798767f1"
server_key_version: "encryptionkey@0"
I20260927 03:23:44.864208 9593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:44.864986 9607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:44.865159 9593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:44.865258 9593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "286a40b4f3384139918cbc33892d556e"
format_stamp: "Formatted at 2026-09-27 03:23:44 on dist-test-slave-gw2v"
server_key: "eff1c9bb2a20d48ef3b3e439702e1717"
server_key_iv: "cc75cb96333c61d15cb5b118798767f1"
server_key_version: "encryptionkey@0"
I20260927 03:23:44.865324 9593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:44.907591 9593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:44.907914 9593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:44.908043 9593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:44.911805 9593 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34045
I20260927 03:23:44.911841 9659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34045 every 8 connection(s)
I20260927 03:23:44.912185 9593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:44.912822 9660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:44.915499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 9593
I20260927 03:23:44.915591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:44.915577 9660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e: Bootstrap starting.
I20260927 03:23:44.915750 392 external_mini_cluster.cc:1428] Setting key c5dbe391000afea4d999ce135a043d3d
I20260927 03:23:44.916239 9660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:44.916520 9660 log.cc:824] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:44.917203 9660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e: No bootstrap required, opened a new log
I20260927 03:23:44.919237 9660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286a40b4f3384139918cbc33892d556e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34045 } }
I20260927 03:23:44.919406 9660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:44.919450 9660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286a40b4f3384139918cbc33892d556e, State: Initialized, Role: FOLLOWER
I20260927 03:23:44.919580 9660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286a40b4f3384139918cbc33892d556e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34045 } }
I20260927 03:23:44.919668 9660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:44.919703 9660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:44.919744 9660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:44.920394 9660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286a40b4f3384139918cbc33892d556e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34045 } }
I20260927 03:23:44.920500 9660 leader_election.cc:302] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 286a40b4f3384139918cbc33892d556e; no voters:
I20260927 03:23:44.920644 9660 leader_election.cc:288] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:44.920785 9665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:44.920919 9660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:44.920953 9665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [term 1 LEADER]: Becoming Leader. State: Replica: 286a40b4f3384139918cbc33892d556e, State: Running, Role: LEADER
I20260927 03:23:44.921027 9665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286a40b4f3384139918cbc33892d556e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34045 } }
I20260927 03:23:44.921365 9665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 286a40b4f3384139918cbc33892d556e. Latest consensus state: current_term: 1 leader_uuid: "286a40b4f3384139918cbc33892d556e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286a40b4f3384139918cbc33892d556e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34045 } } }
I20260927 03:23:44.921525 9665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:44.921602 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:44.916240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56364 (local address 127.0.98.62:34045)
0927 03:23:44.916430 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:44.916438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:44.916461 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:23:44.916991 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:44.916999 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:44.917063 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:44.917215 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:44.917447 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:44.918377 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:44.919520 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:44.920101 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:44.920219 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:44.920889 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:44.920893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:44.920894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:44.921085 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:44.921129 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:44.921144 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:44.921183 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:44.921314 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 03:23:44.921407 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260927 03:23:44.921717 9674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:44.922195 9674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:44.922071 9666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "286a40b4f3384139918cbc33892d556e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286a40b4f3384139918cbc33892d556e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34045 } } }
I20260927 03:23:44.922530 9666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:44.923688 9674 catalog_manager.cc:1347] Generated new cluster ID: 1527900cf5bf4b5f8e359a84b868ea9f
I20260927 03:23:44.923735 9674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:23:44.934695 9674 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:44.935256 9674 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:44.941886 9674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 286a40b4f3384139918cbc33892d556e: Generated new TSK 0
I20260927 03:23:44.942080 9674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:23:44.951808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34045
--builtin_ntp_servers=127.0.98.20:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 03:23:45.038908 9684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:45.039126 9684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:45.039152 9684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:45.040618 9684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:45.040684 9684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:45.040724 9684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:23:45.042130 9684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:45.042405 9684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:45.042719 9684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:45.047945 9692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:45.047983 9689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:45.047983 9690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:45.048236 9684 server_base.cc:1034] running on GCE node
I20260927 03:23:45.048493 9684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:45.048746 9684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:45.049904 9684 hybrid_clock.cc:648] HybridClock initialized: now 1790479425049889 us; error 30 us; skew 500 ppm
I20260927 03:23:45.051255 9684 webserver.cc:460] Webserver started at http://127.0.98.1:40059/ using document root <none> and password file <none>
I20260927 03:23:45.051501 9684 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:45.051544 9684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:45.051659 9684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:45.052724 9684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "1e945ddb3d6f49da92976006b6254d9b"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "12a9f626ee62c8f051f65b230bb3d332"
server_key_iv: "b15645652f5dc5c9bb0428171495f2de"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.053047 9684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1e945ddb3d6f49da92976006b6254d9b"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "12a9f626ee62c8f051f65b230bb3d332"
server_key_iv: "b15645652f5dc5c9bb0428171495f2de"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.054389 9684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:45.055286 9698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.055461 9684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:23:45.055600 9684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "1e945ddb3d6f49da92976006b6254d9b"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "12a9f626ee62c8f051f65b230bb3d332"
server_key_iv: "b15645652f5dc5c9bb0428171495f2de"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.055665 9684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:45.068397 9684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:45.068688 9684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:45.068794 9684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:45.069018 9684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:45.069358 9684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:45.069392 9684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.069413 9684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:45.069424 9684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.074707 9684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32887
I20260927 03:23:45.074779 9811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32887 every 8 connection(s)
I20260927 03:23:45.075089 9684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:23:45.077275 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 9684
I20260927 03:23:45.077373 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:23:45.077593 392 external_mini_cluster.cc:1428] Setting key 3883dc0cc448e2da7bdc71092199f918
I20260927 03:23:45.078632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34045
--builtin_ntp_servers=127.0.98.20:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 03:23:45.081040 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:45.075869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59355 (local address 127.0.98.62:34045)
0927 03:23:45.075949 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:45.075952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:45.076127 (+ 175us) server_negotiation.cc:407] Connection header received
0927 03:23:45.076660 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:45.076661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:45.076671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:45.076753 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:45.077632 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:45.078249 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:45.079442 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:45.079743 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:45.079809 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:45.080821 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:45.080822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:45.080822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:45.080827 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:45.080841 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:45.080844 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:45.080884 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:45.080953 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 03:23:45.080973 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 03:23:45.081272 9814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:45.075832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34045 (local address 127.0.98.1:59355)
0927 03:23:45.076010 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:45.076046 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:45.076569 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:45.076810 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:45.076814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:45.076822 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:45.077467 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:45.077475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:45.078336 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:45.078337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:45.079346 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:45.079350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:45.079850 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:45.079851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:45.080055 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:45.080671 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:23:45.080672 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:45.080709 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:23:45.080736 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:45.080887 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:45.080891 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:45.080894 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:45.080955 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:23:45.081064 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260927 03:23:45.081776 9812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34045
I20260927 03:23:45.081909 9812 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:45.082122 9812 heartbeater.cc:511] Master 127.0.98.62:34045 requested a full tablet report, sending...
I20260927 03:23:45.082679 9624 ts_manager.cc:194] Registered new tserver with Master: 1e945ddb3d6f49da92976006b6254d9b (127.0.98.1:32887)
I20260927 03:23:45.083335 9624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:59355
W20260927 03:23:45.157894 9815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:45.158144 9815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:45.158164 9815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:45.159463 9815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:45.159504 9815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:45.159536 9815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:23:45.160833 9815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:45.161049 9815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:45.161321 9815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:45.166306 9820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:45.166334 9823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:45.166334 9821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:45.166469 9815 server_base.cc:1034] running on GCE node
I20260927 03:23:45.166930 9815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:45.167165 9815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:45.168303 9815 hybrid_clock.cc:648] HybridClock initialized: now 1790479425168284 us; error 30 us; skew 500 ppm
I20260927 03:23:45.169696 9815 webserver.cc:460] Webserver started at http://127.0.98.2:43803/ using document root <none> and password file <none>
I20260927 03:23:45.169921 9815 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:45.169972 9815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:45.170079 9815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:45.171214 9815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "1d20f7cc35874605866a2f4aea11feb6"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "1ecde45a6e3688fa7a89dc633de7d687"
server_key_iv: "37086428161712c07b4fe09c1a0bbd04"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.171516 9815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1d20f7cc35874605866a2f4aea11feb6"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "1ecde45a6e3688fa7a89dc633de7d687"
server_key_iv: "37086428161712c07b4fe09c1a0bbd04"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.172734 9815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:23:45.173589 9829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.173748 9815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:45.173830 9815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "1d20f7cc35874605866a2f4aea11feb6"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "1ecde45a6e3688fa7a89dc633de7d687"
server_key_iv: "37086428161712c07b4fe09c1a0bbd04"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.173892 9815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:45.181661 9815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:45.181916 9815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:45.182029 9815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:45.182212 9815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:45.182528 9815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:45.182561 9815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.182588 9815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:45.182613 9815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.187996 9815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45677
I20260927 03:23:45.188033 9942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45677 every 8 connection(s)
I20260927 03:23:45.188338 9815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:23:45.193367 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:45.189067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48769 (local address 127.0.98.62:34045)
0927 03:23:45.189150 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:45.189152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:45.189290 (+ 138us) server_negotiation.cc:407] Connection header received
0927 03:23:45.189807 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:45.189808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:45.189820 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:45.189874 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:45.190608 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:45.191146 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:45.192114 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:45.192295 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:45.192348 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:45.193113 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:45.193114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:45.193115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:45.193130 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:45.193145 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:45.193148 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:45.193190 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:45.193261 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 03:23:45.193285 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 03:23:45.193485 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 9815
I20260927 03:23:45.193571 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:23:45.193639 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:45.188975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34045 (local address 127.0.98.2:48769)
0927 03:23:45.189176 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:45.189212 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:45.189714 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:45.189949 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:45.189952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:45.189960 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:45.190516 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:45.190521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:45.191229 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:45.191230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:45.192038 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:45.192040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:45.192359 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:45.192360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:45.192493 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:45.192984 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:23:45.192984 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:45.193018 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:23:45.193041 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:45.193195 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:45.193200 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:45.193203 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:45.193261 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:23:45.193358 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260927 03:23:45.193764 392 external_mini_cluster.cc:1428] Setting key 34e7ce70441ca2d050a3f64917cdfcad
I20260927 03:23:45.194079 9943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34045
I20260927 03:23:45.194157 9943 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:45.194339 9943 heartbeater.cc:511] Master 127.0.98.62:34045 requested a full tablet report, sending...
I20260927 03:23:45.194705 9624 ts_manager.cc:194] Registered new tserver with Master: 1d20f7cc35874605866a2f4aea11feb6 (127.0.98.2:45677)
I20260927 03:23:45.194702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34045
--builtin_ntp_servers=127.0.98.20:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 03:23:45.195199 9624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48769
W20260927 03:23:45.273799 9946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:45.274034 9946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:45.274066 9946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:45.275295 9946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:45.275339 9946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:45.275373 9946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:23:45.276762 9946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:45.277007 9946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:45.277287 9946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:45.282354 9952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:45.282354 9951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:45.282464 9946 server_base.cc:1034] running on GCE node
W20260927 03:23:45.282647 9954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:45.282874 9946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:45.283169 9946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:45.284325 9946 hybrid_clock.cc:648] HybridClock initialized: now 1790479425284308 us; error 35 us; skew 500 ppm
I20260927 03:23:45.285601 9946 webserver.cc:460] Webserver started at http://127.0.98.3:33983/ using document root <none> and password file <none>
I20260927 03:23:45.285817 9946 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:45.285866 9946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:45.286023 9946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:45.287134 9946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "421a03cd51c94119afbf62ed924de314"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "2a1e736af2884e929c860d9bd3594470"
server_key_iv: "bae61db224523957cbf02bcc69302e85"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.287461 9946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "421a03cd51c94119afbf62ed924de314"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "2a1e736af2884e929c860d9bd3594470"
server_key_iv: "bae61db224523957cbf02bcc69302e85"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.288667 9946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:23:45.289530 9960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.289724 9946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:23:45.289811 9946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "421a03cd51c94119afbf62ed924de314"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "2a1e736af2884e929c860d9bd3594470"
server_key_iv: "bae61db224523957cbf02bcc69302e85"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.289877 9946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:45.323485 9946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:45.323799 9946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:45.323923 9946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:45.324151 9946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:45.324455 9946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:45.324488 9946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.324517 9946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:45.324540 9946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.329795 9946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44337
I20260927 03:23:45.329838 10073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44337 every 8 connection(s)
I20260927 03:23:45.330154 9946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:23:45.335289 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:45.330861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38079 (local address 127.0.98.62:34045)
0927 03:23:45.330916 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:45.330918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:45.331140 (+ 222us) server_negotiation.cc:407] Connection header received
0927 03:23:45.331650 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:45.331652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:45.331664 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:45.331722 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:45.332373 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:45.332873 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:45.333914 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:45.334118 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:45.334175 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:45.335009 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:45.335011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:45.335011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:45.335016 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:45.335033 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:45.335037 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:45.335080 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:45.335173 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 03:23:45.335202 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:23:45.335493 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:45.330811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34045 (local address 127.0.98.3:38079)
0927 03:23:45.331019 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:45.331059 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:45.331557 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:45.331794 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:45.331798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:45.331807 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:45.332286 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:45.332291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:45.332959 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:45.332960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:45.333829 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:45.333831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:45.334188 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:45.334189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:45.334318 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:45.334878 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:23:45.334879 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:45.334910 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:23:45.334936 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:45.335103 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:45.335109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:45.335113 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:45.335172 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 03:23:45.335279 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260927 03:23:45.335991 10074 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34045
I20260927 03:23:45.336078 10074 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:45.336231 10074 heartbeater.cc:511] Master 127.0.98.62:34045 requested a full tablet report, sending...
I20260927 03:23:45.336594 9624 ts_manager.cc:194] Registered new tserver with Master: 421a03cd51c94119afbf62ed924de314 (127.0.98.3:44337)
I20260927 03:23:45.336928 9624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38079
I20260927 03:23:45.339603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 9946
I20260927 03:23:45.339676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:23:45.339808 392 external_mini_cluster.cc:1428] Setting key 00345940d8a264b8b6ac27b1f9736e5a
I20260927 03:23:45.340603 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 03:23:45.342402 9663 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:45.341832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.0.98.62:34045)
0927 03:23:45.341937 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:45.341939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:45.342028 (+ 89us) server_negotiation.cc:407] Connection header received
0927 03:23:45.342102 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:45.342103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:45.342115 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:45.342173 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:45.342334 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56370: BlockingRecv error: recv got EOF from 127.0.0.1:56370 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 03:23:45.342451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 9684
I20260927 03:23:45.346962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 9815
I20260927 03:23:45.350667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 9946
I20260927 03:23:45.354312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 9593
2026-09-27T03:23:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (607 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T03:23:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:45Z Disabled control of system clock
I20260927 03:23:45.378453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41489
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:23:45.458841 10085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:45.459055 10085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:45.459084 10085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:45.460191 10085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:45.460225 10085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:45.460237 10085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:45.460247 10085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:45.460258 10085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:45.461570 10085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:45.461769 10085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:45.462013 10085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:45.466595 10093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:45.466661 10090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:45.466640 10091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:45.466809 10085 server_base.cc:1034] running on GCE node
I20260927 03:23:45.467105 10085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:45.467368 10085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:45.468516 10085 hybrid_clock.cc:648] HybridClock initialized: now 1790479425468501 us; error 33 us; skew 500 ppm
I20260927 03:23:45.469764 10085 webserver.cc:460] Webserver started at http://127.0.98.62:36011/ using document root <none> and password file <none>
I20260927 03:23:45.469973 10085 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:45.470019 10085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:45.470124 10085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:45.471216 10085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "7800557503d64d95b09a94bfa581f091"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "b1db8a8dc2e0489db3b2a89609e024cc"
server_key_iv: "169488f5d52676da75665263edc0fb86"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.471537 10085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "7800557503d64d95b09a94bfa581f091"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "b1db8a8dc2e0489db3b2a89609e024cc"
server_key_iv: "169488f5d52676da75665263edc0fb86"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.472764 10085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 03:23:45.473405 10099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.473531 10085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:45.473619 10085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "7800557503d64d95b09a94bfa581f091"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "b1db8a8dc2e0489db3b2a89609e024cc"
server_key_iv: "169488f5d52676da75665263edc0fb86"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.473676 10085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:45.476953 10085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:45.477193 10085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:45.477310 10085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:45.480932 10085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41489
I20260927 03:23:45.480959 10151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41489 every 8 connection(s)
I20260927 03:23:45.481246 10085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:45.481851 10152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:45.482689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 10085
I20260927 03:23:45.482767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:45.482944 392 external_mini_cluster.cc:1428] Setting key 9bf1a0a7e8ca62b7999882bc23ca0ee6
I20260927 03:23:45.484911 10152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091: Bootstrap starting.
I20260927 03:23:45.485512 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:45.483443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38184 (local address 127.0.98.62:41489)
0927 03:23:45.483628 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:45.483638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:45.483707 (+ 69us) server_negotiation.cc:407] Connection header received
0927 03:23:45.484217 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:45.484225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:45.484267 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:45.484393 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:45.485118 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:45.485121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:45.485122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:45.485126 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:45.485165 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:45.485175 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:45.485253 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:45.485306 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 03:23:45.485362 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260927 03:23:45.485612 10152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:45.485929 10152 log.cc:824] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:45.486644 10152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091: No bootstrap required, opened a new log
I20260927 03:23:45.488214 10152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7800557503d64d95b09a94bfa581f091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } }
I20260927 03:23:45.488349 10152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:45.488370 10152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7800557503d64d95b09a94bfa581f091, State: Initialized, Role: FOLLOWER
I20260927 03:23:45.488452 10152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7800557503d64d95b09a94bfa581f091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } }
I20260927 03:23:45.488509 10152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:45.488529 10152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:45.488564 10152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:45.489056 10152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7800557503d64d95b09a94bfa581f091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } }
I20260927 03:23:45.489137 10152 leader_election.cc:302] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7800557503d64d95b09a94bfa581f091; no voters:
I20260927 03:23:45.489259 10152 leader_election.cc:288] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:45.489490 10152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:45.489562 10157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:45.489730 10157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [term 1 LEADER]: Becoming Leader. State: Replica: 7800557503d64d95b09a94bfa581f091, State: Running, Role: LEADER
I20260927 03:23:45.489838 10157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7800557503d64d95b09a94bfa581f091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } }
I20260927 03:23:45.490222 10158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7800557503d64d95b09a94bfa581f091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7800557503d64d95b09a94bfa581f091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } }
I20260927 03:23:45.490247 10159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7800557503d64d95b09a94bfa581f091. Latest consensus state: current_term: 1 leader_uuid: "7800557503d64d95b09a94bfa581f091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7800557503d64d95b09a94bfa581f091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } }
I20260927 03:23:45.490314 10158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:45.490317 10159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091 [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:45.490502 10167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:45.490967 10167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:45.492441 10167 catalog_manager.cc:1347] Generated new cluster ID: 47b5c0bb9a34485ebe8f7133b5daf221
I20260927 03:23:45.492493 10167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:23:45.507418 10167 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:45.507584 10167 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:45.513579 10167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7800557503d64d95b09a94bfa581f091: Generated new TSK 0
I20260927 03:23:45.513773 10167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:23:45.521189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41489
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:23:45.601979 10176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:45.602196 10176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:45.602219 10176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:45.603406 10176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:45.603452 10176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:45.603479 10176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:23:45.604807 10176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:45.605044 10176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:45.605340 10176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:45.611047 10184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:45.611063 10182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:45.611073 10181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:45.611459 10176 server_base.cc:1034] running on GCE node
I20260927 03:23:45.611698 10176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:45.611974 10176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:45.613150 10176 hybrid_clock.cc:648] HybridClock initialized: now 1790479425613133 us; error 30 us; skew 500 ppm
I20260927 03:23:45.614423 10176 webserver.cc:460] Webserver started at http://127.0.98.1:34523/ using document root <none> and password file <none>
I20260927 03:23:45.614658 10176 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:45.614707 10176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:45.614857 10176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:45.615978 10176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "31052bcbfedc4432932216f1296941ef"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "4323530481326890814c02dcca9f4577"
server_key_iv: "7a49a7e68a472bb7df7e9205db1bbcdc"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.616302 10176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "31052bcbfedc4432932216f1296941ef"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "4323530481326890814c02dcca9f4577"
server_key_iv: "7a49a7e68a472bb7df7e9205db1bbcdc"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.617542 10176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:23:45.618155 10190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.618311 10176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:23:45.618391 10176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "31052bcbfedc4432932216f1296941ef"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "4323530481326890814c02dcca9f4577"
server_key_iv: "7a49a7e68a472bb7df7e9205db1bbcdc"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.618453 10176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:45.621871 10176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:45.622119 10176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:45.622236 10176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:45.622426 10176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:45.622709 10176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:45.622740 10176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.622764 10176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:45.622788 10176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.628118 10176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36335
I20260927 03:23:45.628158 10303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36335 every 8 connection(s)
I20260927 03:23:45.628449 10176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:23:45.631218 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:45.629048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50167 (local address 127.0.98.62:41489)
0927 03:23:45.629090 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:45.629092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:45.629344 (+ 252us) server_negotiation.cc:407] Connection header received
0927 03:23:45.629809 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:45.629810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:45.629817 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:45.629871 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:45.630982 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:45.630984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:45.630984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:45.630989 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:45.631004 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:45.631009 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:45.631059 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:45.631126 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 03:23:45.631146 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:23:45.631395 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:45.629057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41489 (local address 127.0.98.1:50167)
0927 03:23:45.629223 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:45.629255 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:45.629712 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:45.629949 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:45.629953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:45.629963 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:45.630795 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:23:45.630796 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:45.630845 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:23:45.630891 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:45.631062 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:45.631067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:45.631070 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:45.631137 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:23:45.631198 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260927 03:23:45.631832 10304 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41489
I20260927 03:23:45.631912 10304 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:45.632045 10304 heartbeater.cc:511] Master 127.0.98.62:41489 requested a full tablet report, sending...
I20260927 03:23:45.632504 10116 ts_manager.cc:194] Registered new tserver with Master: 31052bcbfedc4432932216f1296941ef (127.0.98.1:36335)
I20260927 03:23:45.635973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 10176
I20260927 03:23:45.636044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:23:45.636173 392 external_mini_cluster.cc:1428] Setting key 6909792eab1842baab6628f6e0b56f5d
I20260927 03:23:45.636938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41489
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:23:45.721132 10307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:45.721343 10307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:45.721381 10307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:45.722601 10307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:45.722640 10307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:45.722674 10307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:23:45.724053 10307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:45.724280 10307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:45.724555 10307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:45.729288 10315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:45.729316 10313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:45.729314 10312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:45.729559 10307 server_base.cc:1034] running on GCE node
I20260927 03:23:45.729760 10307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:45.729965 10307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:45.731092 10307 hybrid_clock.cc:648] HybridClock initialized: now 1790479425731081 us; error 27 us; skew 500 ppm
I20260927 03:23:45.732332 10307 webserver.cc:460] Webserver started at http://127.0.98.2:35479/ using document root <none> and password file <none>
I20260927 03:23:45.732543 10307 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:45.732586 10307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:45.732683 10307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:45.733707 10307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "752479cd9efe438bbd03b36dd790dfb0"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "a4b5447cd433e701ba507f2c7733db19"
server_key_iv: "60e18466d941f2684255bd16e2158eb5"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.733995 10307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "752479cd9efe438bbd03b36dd790dfb0"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "a4b5447cd433e701ba507f2c7733db19"
server_key_iv: "60e18466d941f2684255bd16e2158eb5"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.735157 10307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:45.735824 10321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.735975 10307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:45.736055 10307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "752479cd9efe438bbd03b36dd790dfb0"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "a4b5447cd433e701ba507f2c7733db19"
server_key_iv: "60e18466d941f2684255bd16e2158eb5"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.736116 10307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:45.739310 10307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:45.739549 10307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:45.739666 10307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:45.739842 10307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:45.740113 10307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:45.740145 10307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.740170 10307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:45.740192 10307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.745605 10307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34873
I20260927 03:23:45.745641 10434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34873 every 8 connection(s)
I20260927 03:23:45.745961 10307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:23:45.749074 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:45.746683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36385 (local address 127.0.98.62:41489)
0927 03:23:45.746753 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:45.746755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:45.747038 (+ 283us) server_negotiation.cc:407] Connection header received
0927 03:23:45.747569 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:45.747570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:45.747577 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:45.747634 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:45.748834 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:45.748836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:45.748836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:45.748841 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:45.748857 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:45.748862 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:45.748945 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:45.748986 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 03:23:45.749001 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:23:45.749264 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:45.746632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41489 (local address 127.0.98.2:36385)
0927 03:23:45.746914 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:45.746950 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:45.747453 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:45.747714 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:45.747718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:45.747728 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:45.748669 (+ 941us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:23:45.748670 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:45.748700 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:23:45.748731 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:45.748913 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:45.748917 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:45.748921 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:45.748997 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 03:23:45.749063 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
I20260927 03:23:45.749723 10435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41489
I20260927 03:23:45.749814 10435 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:45.749959 10435 heartbeater.cc:511] Master 127.0.98.62:41489 requested a full tablet report, sending...
I20260927 03:23:45.750247 10116 ts_manager.cc:194] Registered new tserver with Master: 752479cd9efe438bbd03b36dd790dfb0 (127.0.98.2:34873)
I20260927 03:23:45.752555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 10307
I20260927 03:23:45.752629 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:23:45.752767 392 external_mini_cluster.cc:1428] Setting key 8e9f6e56fe19cd2b907a55065d19f133
I20260927 03:23:45.753614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41489
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:23:45.832145 10438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:45.832350 10438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:45.832369 10438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:45.833529 10438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:45.833567 10438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:45.833591 10438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:23:45.834909 10438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:45.835094 10438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:45.835333 10438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:45.840196 10446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:45.840291 10438 server_base.cc:1034] running on GCE node
W20260927 03:23:45.840196 10443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:45.840212 10444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:45.840611 10438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:45.840819 10438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:45.841944 10438 hybrid_clock.cc:648] HybridClock initialized: now 1790479425841933 us; error 27 us; skew 500 ppm
I20260927 03:23:45.843202 10438 webserver.cc:460] Webserver started at http://127.0.98.3:41469/ using document root <none> and password file <none>
I20260927 03:23:45.843425 10438 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:45.843472 10438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:45.843601 10438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:45.844667 10438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "a5e99a7737c0402680172abed4bea704"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "2adc5a05cce9fdc8ee68520cc992b61d"
server_key_iv: "0ce26bd76f65ee36c6cddb09334f2543"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.844991 10438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a5e99a7737c0402680172abed4bea704"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "2adc5a05cce9fdc8ee68520cc992b61d"
server_key_iv: "0ce26bd76f65ee36c6cddb09334f2543"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.846199 10438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:23:45.846896 10452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.847033 10438 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.847110 10438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "a5e99a7737c0402680172abed4bea704"
format_stamp: "Formatted at 2026-09-27 03:23:45 on dist-test-slave-gw2v"
server_key: "2adc5a05cce9fdc8ee68520cc992b61d"
server_key_iv: "0ce26bd76f65ee36c6cddb09334f2543"
server_key_version: "encryptionkey@0"
I20260927 03:23:45.847172 10438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:45.850476 10438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:45.850739 10438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:45.850888 10438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:45.851081 10438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:45.851361 10438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:45.851392 10438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.851418 10438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:45.851442 10438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:45.856930 10438 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43439
I20260927 03:23:45.857023 10565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43439 every 8 connection(s)
I20260927 03:23:45.857300 10438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:23:45.857816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 10438
I20260927 03:23:45.857889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:23:45.858054 392 external_mini_cluster.cc:1428] Setting key 00f6702fe6c3d7e2c4427826e3b89c37
I20260927 03:23:45.860720 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:45.858550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60473 (local address 127.0.98.62:41489)
0927 03:23:45.858599 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:45.858601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:45.858913 (+ 312us) server_negotiation.cc:407] Connection header received
0927 03:23:45.859389 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:45.859390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:45.859396 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:45.859453 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:45.860525 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:45.860526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:45.860526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:23:45.860530 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:45.860544 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:23:45.860549 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:45.860601 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:45.860654 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 03:23:45.860667 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:23:45.860881 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:45.858569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41489 (local address 127.0.98.3:60473)
0927 03:23:45.858766 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:45.858809 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:45.859301 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:45.859518 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:45.859521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:45.859529 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:45.860380 (+ 851us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:23:45.860380 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:45.860409 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:23:45.860437 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:45.860598 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:45.860601 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:45.860603 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:23:45.860652 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:23:45.860700 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260927 03:23:45.861263 10566 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41489
I20260927 03:23:45.861395 10566 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:45.861547 10566 heartbeater.cc:511] Master 127.0.98.62:41489 requested a full tablet report, sending...
I20260927 03:23:45.861855 10116 ts_manager.cc:194] Registered new tserver with Master: a5e99a7737c0402680172abed4bea704 (127.0.98.3:43439)
I20260927 03:23:45.869756 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:23:45.871929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 10176
W20260927 03:23:45.871961 10155 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:23:45.871203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38200 (local address 127.0.98.62:41489)
0927 03:23:45.871265 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:45.871268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:45.871360 (+ 92us) server_negotiation.cc:407] Connection header received
0927 03:23:45.871434 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:45.871435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:45.871440 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:45.871549 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:45.871859 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38200: BlockingRecv error: recv got EOF from 127.0.0.1:38200 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:23:45.875978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 10307
I20260927 03:23:45.879709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 10438
I20260927 03:23:45.883380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 10085
2026-09-27T03:23:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (529 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:23:45 dist-test-slave-gw2v krb5kdc[10576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:23:45 dist-test-slave-gw2v krb5kdc[10576](info): set up 2 sockets
Sep 27 03:23:45 dist-test-slave-gw2v krb5kdc[10576](info): commencing operation
krb5kdc: starting...
W20260927 03:23:46.843612 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:23:46 dist-test-slave-gw2v krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:23:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:23:46.961627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39419
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:47.042963 10589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:47.043246 10589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:47.043278 10589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:47.044463 10589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:23:47.044512 10589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:47.044533 10589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:47.044551 10589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:23:47.044571 10589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:23:47.046082 10589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:47.046334 10589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:47.046545 10589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:47.051210 10595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:47.051306 10594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:47.051330 10597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:47.051486 10589 server_base.cc:1034] running on GCE node
I20260927 03:23:47.051851 10589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:47.052125 10589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:47.053324 10589 hybrid_clock.cc:648] HybridClock initialized: now 1790479427053311 us; error 28 us; skew 500 ppm
Sep 27 03:23:47 dist-test-slave-gw2v krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:47.055845 10589 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:23:47.056347 10589 webserver.cc:460] Webserver started at http://127.0.98.62:37949/ using document root <none> and password file <none>
I20260927 03:23:47.056561 10589 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:47.056612 10589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:47.056740 10589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:47.057786 10589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "3326cf3e8ab7448ca024286ceec4cd0d"
format_stamp: "Formatted at 2026-09-27 03:23:47 on dist-test-slave-gw2v"
server_key: "714a32011eda9ea82e54673ba8f9b967"
server_key_iv: "99169515cdb5f2bfdd55a0249bb2b04b"
server_key_version: "encryptionkey@0"
I20260927 03:23:47.058127 10589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "3326cf3e8ab7448ca024286ceec4cd0d"
format_stamp: "Formatted at 2026-09-27 03:23:47 on dist-test-slave-gw2v"
server_key: "714a32011eda9ea82e54673ba8f9b967"
server_key_iv: "99169515cdb5f2bfdd55a0249bb2b04b"
server_key_version: "encryptionkey@0"
I20260927 03:23:47.059517 10589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:23:47.060185 10604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:47.060328 10589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:47.060415 10589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "3326cf3e8ab7448ca024286ceec4cd0d"
format_stamp: "Formatted at 2026-09-27 03:23:47 on dist-test-slave-gw2v"
server_key: "714a32011eda9ea82e54673ba8f9b967"
server_key_iv: "99169515cdb5f2bfdd55a0249bb2b04b"
server_key_version: "encryptionkey@0"
I20260927 03:23:47.060479 10589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:47.075096 10589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:47.077205 10589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:47.077315 10589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:47.081071 10589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39419
I20260927 03:23:47.081112 10656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39419 every 8 connection(s)
I20260927 03:23:47.081418 10589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:23:47.082427 10657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:47.084681 10657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d: Bootstrap starting.
I20260927 03:23:47.085215 10657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:47.085477 10657 log.cc:824] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:47.086176 10657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d: No bootstrap required, opened a new log
I20260927 03:23:47.086491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 10589
I20260927 03:23:47.086576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:23:47.086725 392 external_mini_cluster.cc:1428] Setting key 5b60182b34f0b482047e4d1182d3934d
I20260927 03:23:47.088457 10657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3326cf3e8ab7448ca024286ceec4cd0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39419 } }
I20260927 03:23:47.088644 10657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:47.088691 10657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3326cf3e8ab7448ca024286ceec4cd0d, State: Initialized, Role: FOLLOWER
I20260927 03:23:47.088827 10657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3326cf3e8ab7448ca024286ceec4cd0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39419 } }
I20260927 03:23:47.088919 10657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:23:47.088963 10657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:23:47.089015 10657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:47.089577 10657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3326cf3e8ab7448ca024286ceec4cd0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39419 } }
I20260927 03:23:47.089677 10657 leader_election.cc:302] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3326cf3e8ab7448ca024286ceec4cd0d; no voters:
I20260927 03:23:47.089835 10657 leader_election.cc:288] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:23:47.089870 10662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:47.089982 10662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [term 1 LEADER]: Becoming Leader. State: Replica: 3326cf3e8ab7448ca024286ceec4cd0d, State: Running, Role: LEADER
I20260927 03:23:47.090104 10657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:23:47.090137 10662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3326cf3e8ab7448ca024286ceec4cd0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39419 } }
I20260927 03:23:47.090476 10662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3326cf3e8ab7448ca024286ceec4cd0d. Latest consensus state: current_term: 1 leader_uuid: "3326cf3e8ab7448ca024286ceec4cd0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3326cf3e8ab7448ca024286ceec4cd0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39419 } } }
I20260927 03:23:47.090731 10662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [sys.catalog]: This master's current role is: LEADER
I20260927 03:23:47.091158 10671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:23:47.091621 10671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:23:47.090646 10663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3326cf3e8ab7448ca024286ceec4cd0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3326cf3e8ab7448ca024286ceec4cd0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39419 } } }
I20260927 03:23:47.092235 10663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d [sys.catalog]: This master's current role is: LEADER
Sep 27 03:23:47 dist-test-slave-gw2v krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:47.095355 10671 catalog_manager.cc:1347] Generated new cluster ID: 6f2184544aad48f5900c2903387b7271
I20260927 03:23:47.095404 10671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:23:47.098286 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.087241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41148 (local address 127.0.98.62:39419)
0927 03:23:47.087413 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:47.087421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:47.087446 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:23:47.087946 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:47.087952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:47.087999 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:47.088111 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:47.088910 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.090181 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.091262 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.091864 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.091994 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.094592 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:47.094604 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:47.094604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:47.094615 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:47.096619 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:47.097023 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:47.097026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:47.097026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:47.097124 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:47.097331 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:47.097331 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:47.097332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:47.097609 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:47.097742 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:47.097861 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 03:23:47.097982 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260927 03:23:47.115587 10671 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:23:47.116129 10671 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:23:47.121727 10671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d: Generated new TSK 0
I20260927 03:23:47.121996 10671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:23:47.149250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39419
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:47.229236 10683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:47.229451 10683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:47.229476 10683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:47.230558 10683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:47.230593 10683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:47.230618 10683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:23:47.232046 10683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:47.232232 10683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:47.232421 10683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:47.237223 10691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:47.237223 10689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:47.237218 10688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:47.237542 10683 server_base.cc:1034] running on GCE node
I20260927 03:23:47.237774 10683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:47.238027 10683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:47.239202 10683 hybrid_clock.cc:648] HybridClock initialized: now 1790479427239183 us; error 27 us; skew 500 ppm
Sep 27 03:23:47 dist-test-slave-gw2v krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:47.241779 10683 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:23:47.242209 10683 webserver.cc:460] Webserver started at http://127.0.98.1:37195/ using document root <none> and password file <none>
I20260927 03:23:47.242468 10683 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:47.242520 10683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:47.242642 10683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:47.243748 10683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "6693b463eb0b43f2bec39cc0c4a5917e"
format_stamp: "Formatted at 2026-09-27 03:23:47 on dist-test-slave-gw2v"
server_key: "a7561b7a2359a10f086c460e03176193"
server_key_iv: "aa89484ce9bfe654ba422ce1465d4fff"
server_key_version: "encryptionkey@0"
I20260927 03:23:47.244074 10683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6693b463eb0b43f2bec39cc0c4a5917e"
format_stamp: "Formatted at 2026-09-27 03:23:47 on dist-test-slave-gw2v"
server_key: "a7561b7a2359a10f086c460e03176193"
server_key_iv: "aa89484ce9bfe654ba422ce1465d4fff"
server_key_version: "encryptionkey@0"
I20260927 03:23:47.245407 10683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:23:47.246101 10698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:47.246307 10683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:47.246389 10683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "6693b463eb0b43f2bec39cc0c4a5917e"
format_stamp: "Formatted at 2026-09-27 03:23:47 on dist-test-slave-gw2v"
server_key: "a7561b7a2359a10f086c460e03176193"
server_key_iv: "aa89484ce9bfe654ba422ce1465d4fff"
server_key_version: "encryptionkey@0"
I20260927 03:23:47.246455 10683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:47.263206 10683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:47.265115 10683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:47.265256 10683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:47.265475 10683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:47.265767 10683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:47.265801 10683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:47.265830 10683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:47.265852 10683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:47.271350 10683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33605
I20260927 03:23:47.271392 10811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33605 every 8 connection(s)
I20260927 03:23:47.271728 10683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:23:47.273957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 10683
I20260927 03:23:47.274048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:23:47.274199 392 external_mini_cluster.cc:1428] Setting key 8d7c315009738b2522466c24293d4bb9
Sep 27 03:23:47 dist-test-slave-gw2v krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:47.281766 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.272506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48969 (local address 127.0.98.62:39419)
0927 03:23:47.272585 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:47.272587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:47.272711 (+ 124us) server_negotiation.cc:407] Connection header received
0927 03:23:47.273246 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:47.273247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:47.273260 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:47.273310 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:47.274344 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.274965 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.276351 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.276596 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.276669 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.278851 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:47.278853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:47.278853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:47.278862 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:47.280340 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:47.280776 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:47.280777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:47.280778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:47.280851 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:47.281134 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:47.281135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:47.281135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:47.281274 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:47.281344 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:47.281582 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 03:23:47.281667 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 03:23:47.281879 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.272403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39419 (local address 127.0.98.1:48969)
0927 03:23:47.272585 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:47.272630 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:47.273146 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:47.273381 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:47.273385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:47.273982 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:47.274209 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.274215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.275046 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.275047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.276247 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.276250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.276683 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.276684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.276820 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.277355 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:47.277364 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:47.278724 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:47.280432 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:47.280439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:47.280440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:47.280672 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:47.280950 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:47.280951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:47.280951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:47.281065 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:47.281334 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:47.281337 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:47.281458 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 03:23:47.281537 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 03:23:47.281665 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260927 03:23:47.282392 10812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39419
I20260927 03:23:47.282490 10812 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 03:23:47.282665 10812 heartbeater.cc:511] Master 127.0.98.62:39419 requested a full tablet report, sending...
I20260927 03:23:47.283210 10621 ts_manager.cc:194] Registered new tserver with Master: 6693b463eb0b43f2bec39cc0c4a5917e (127.0.98.1:33605)
I20260927 03:23:47.283865 10621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:23:47.299592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39419
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:47.378578 10817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:47.378795 10817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:47.378837 10817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:47.380009 10817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:47.380054 10817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:47.380080 10817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:23:47.381489 10817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:47.381680 10817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:47.381870 10817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:23:47.386693 10823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:47.386693 10822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:47.386696 10825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:47.386781 10817 server_base.cc:1034] running on GCE node
I20260927 03:23:47.387108 10817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:47.387339 10817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:47.388494 10817 hybrid_clock.cc:648] HybridClock initialized: now 1790479427388473 us; error 38 us; skew 500 ppm
Sep 27 03:23:47 dist-test-slave-gw2v krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:47.391219 10817 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:23:47.391712 10817 webserver.cc:460] Webserver started at http://127.0.98.2:43701/ using document root <none> and password file <none>
I20260927 03:23:47.391945 10817 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:47.391995 10817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:47.392128 10817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:47.393191 10817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "32e8ae455b20488bb8640f934c46bad5"
format_stamp: "Formatted at 2026-09-27 03:23:47 on dist-test-slave-gw2v"
server_key: "066813cca15a92df5f2bc15633e1ec32"
server_key_iv: "fab053b56e93ebcb84ebea21d2098878"
server_key_version: "encryptionkey@0"
I20260927 03:23:47.393561 10817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "32e8ae455b20488bb8640f934c46bad5"
format_stamp: "Formatted at 2026-09-27 03:23:47 on dist-test-slave-gw2v"
server_key: "066813cca15a92df5f2bc15633e1ec32"
server_key_iv: "fab053b56e93ebcb84ebea21d2098878"
server_key_version: "encryptionkey@0"
I20260927 03:23:47.394912 10817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:23:47.395642 10832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:47.395798 10817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:23:47.395880 10817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "32e8ae455b20488bb8640f934c46bad5"
format_stamp: "Formatted at 2026-09-27 03:23:47 on dist-test-slave-gw2v"
server_key: "066813cca15a92df5f2bc15633e1ec32"
server_key_iv: "fab053b56e93ebcb84ebea21d2098878"
server_key_version: "encryptionkey@0"
I20260927 03:23:47.395951 10817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:47.407132 10817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:47.409431 10817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:47.409567 10817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:47.409798 10817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:47.410085 10817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:47.410118 10817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:47.410147 10817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:47.410171 10817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:47.415776 10817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37309
I20260927 03:23:47.415817 10945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37309 every 8 connection(s)
I20260927 03:23:47.416131 10817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:23:47 dist-test-slave-gw2v krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:47.424220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 10817
I20260927 03:23:47.424315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:23:47.424499 392 external_mini_cluster.cc:1428] Setting key 2c4239e68b70b8f57501eb7c19cbc618
I20260927 03:23:47.425801 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.416816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48167 (local address 127.0.98.62:39419)
0927 03:23:47.416906 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:47.416908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:47.417050 (+ 142us) server_negotiation.cc:407] Connection header received
0927 03:23:47.417618 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:47.417619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:47.417628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:47.417681 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:47.418617 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.419147 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.420266 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.420461 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.420516 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.422751 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:47.422753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:47.422753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:47.422759 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:47.424168 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:47.424770 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:47.424771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:47.424771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:47.424827 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:47.425073 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:47.425074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:47.425074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:47.425205 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:47.425265 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:47.425671 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 03:23:47.425737 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:23:47.426143 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.416787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39419 (local address 127.0.98.2:48167)
0927 03:23:47.416931 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:47.416970 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:47.417531 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:47.417739 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:47.417743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:47.418319 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:47.418525 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.418530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.419289 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.419290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.420170 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.420172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.420558 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.420559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.420682 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.421288 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:47.421296 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:47.422667 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:47.424359 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:47.424367 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:47.424370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:47.424702 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:47.424911 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:47.424912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:47.424912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:47.425021 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:47.425327 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:47.425334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:47.425526 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 03:23:47.425661 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 03:23:47.425911 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260927 03:23:47.426764 10946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39419
I20260927 03:23:47.426889 10946 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:47.427069 10946 heartbeater.cc:511] Master 127.0.98.62:39419 requested a full tablet report, sending...
I20260927 03:23:47.427546 10621 ts_manager.cc:194] Registered new tserver with Master: 32e8ae455b20488bb8640f934c46bad5 (127.0.98.2:37309)
I20260927 03:23:47.428021 10621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48167
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:23:47.450676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39419
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:23:47.530002 10951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:23:47.530210 10951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:23:47.530231 10951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:23:47.531409 10951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:23:47.531454 10951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:23:47.531481 10951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:23:47.532912 10951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:23:47.533120 10951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:23:47.533334 10951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:23:47.538111 10951 server_base.cc:1034] running on GCE node
W20260927 03:23:47.538072 10956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:47.538076 10957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:23:47.538076 10959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:23:47.538516 10951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:23:47.538748 10951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:23:47.539901 10951 hybrid_clock.cc:648] HybridClock initialized: now 1790479427539897 us; error 39 us; skew 500 ppm
Sep 27 03:23:47 dist-test-slave-gw2v krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:23:47.542459 10951 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:23:47.542918 10951 webserver.cc:460] Webserver started at http://127.0.98.3:44745/ using document root <none> and password file <none>
I20260927 03:23:47.543105 10951 fs_manager.cc:359] Metadata directory not provided
I20260927 03:23:47.543152 10951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:23:47.543345 10951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:23:47.544431 10951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "caa044013a014946b6d41bdd10f65fba"
format_stamp: "Formatted at 2026-09-27 03:23:47 on dist-test-slave-gw2v"
server_key: "1e2d8983d0bba22cfc5ee6fbe9c10ac1"
server_key_iv: "20ea707f40871a15d9c4212506c95543"
server_key_version: "encryptionkey@0"
I20260927 03:23:47.544760 10951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "caa044013a014946b6d41bdd10f65fba"
format_stamp: "Formatted at 2026-09-27 03:23:47 on dist-test-slave-gw2v"
server_key: "1e2d8983d0bba22cfc5ee6fbe9c10ac1"
server_key_iv: "20ea707f40871a15d9c4212506c95543"
server_key_version: "encryptionkey@0"
I20260927 03:23:47.546159 10951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:23:47.547077 10966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:47.547215 10951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:23:47.547355 10951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "caa044013a014946b6d41bdd10f65fba"
format_stamp: "Formatted at 2026-09-27 03:23:47 on dist-test-slave-gw2v"
server_key: "1e2d8983d0bba22cfc5ee6fbe9c10ac1"
server_key_iv: "20ea707f40871a15d9c4212506c95543"
server_key_version: "encryptionkey@0"
I20260927 03:23:47.547422 10951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:23:47.577857 10951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:23:47.579828 10951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:23:47.579967 10951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:23:47.580186 10951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:23:47.580493 10951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:23:47.580528 10951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:47.580556 10951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:23:47.580578 10951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:23:47.586031 10951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43655
I20260927 03:23:47.586076 11079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43655 every 8 connection(s)
I20260927 03:23:47.586401 10951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:23:47 dist-test-slave-gw2v krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:23:47.595125 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.587122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47433 (local address 127.0.98.62:39419)
0927 03:23:47.587175 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:47.587176 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:47.587369 (+ 193us) server_negotiation.cc:407] Connection header received
0927 03:23:47.587839 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:47.587840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:47.587850 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:47.587898 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:47.588787 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.589267 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.590266 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.590485 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.590536 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.592764 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:47.592765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:47.592766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:47.592772 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:47.593968 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:47.594356 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:47.594356 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:47.594357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:47.594400 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:47.594577 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:47.594577 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:47.594578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:47.594671 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:47.594715 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:47.594997 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 03:23:47.595059 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:23:47.595294 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.587097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39419 (local address 127.0.98.3:47433)
0927 03:23:47.587253 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:47.587289 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:47.587766 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:47.587961 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:47.587964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:47.588514 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:23:47.588718 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.588723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.589341 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.589342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.590191 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.590194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.590544 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.590545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.590647 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.591149 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:23:47.591157 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:23:47.592675 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:23:47.594043 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:47.594049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:47.594051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:47.594297 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:47.594443 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:23:47.594443 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:23:47.594444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:23:47.594536 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:23:47.594722 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:23:47.594729 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:23:47.594883 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 03:23:47.594973 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 03:23:47.595091 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260927 03:23:47.595723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 10951
I20260927 03:23:47.595747 11080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39419
I20260927 03:23:47.595819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:23:47.595831 11080 heartbeater.cc:464] Registering TS with master...
I20260927 03:23:47.596006 11080 heartbeater.cc:511] Master 127.0.98.62:39419 requested a full tablet report, sending...
I20260927 03:23:47.596016 392 external_mini_cluster.cc:1428] Setting key 3407a3a9fa918806d674ccd1c3eb20eb
I20260927 03:23:47.596406 10620 ts_manager.cc:194] Registered new tserver with Master: caa044013a014946b6d41bdd10f65fba (127.0.98.3:43655)
I20260927 03:23:47.596774 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:23:47.596843 10620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47433
I20260927 03:23:47.603662 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.598122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.0.98.62:39419)
0927 03:23:47.598171 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:47.598173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:47.598286 (+ 113us) server_negotiation.cc:407] Connection header received
0927 03:23:47.598348 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:47.598349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:47.598358 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:47.598402 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:23:47.598979 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.599459 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.600189 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.600380 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.600433 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.601487 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:23:47.601489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:23:47.601489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:23:47.601496 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:23:47.602622 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:47.602962 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:47.602963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:47.602963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:47.603017 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:23:47.603173 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:23:47.603174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:23:47.603174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:23:47.603320 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:23:47.603411 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:47.603506 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 03:23:47.603576 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:23:47.605314 10620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:23:47.606805 10620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:23:47.613333 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.609386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47702 (local address 127.0.98.3:43655)
0927 03:23:47.609563 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:47.609568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:47.609576 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:47.609613 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:47.609614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:47.609626 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:47.609692 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:47.609853 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.610688 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.612216 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.612988 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.613073 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.613139 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:47.613207 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 03:23:47.613249 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
I20260927 03:23:47.613305 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.609147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33605 (local address 127.0.0.1:52166)
0927 03:23:47.609550 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:47.609557 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:47.609593 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:47.609798 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:47.609799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:47.609801 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:47.609838 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.609839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.611136 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.611137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.612241 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.612243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.612913 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.612914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.612992 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.613006 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:23:47.613072 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:23:47.613127 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":62,"thread_start_us":60,"threads_started":1}
I20260927 03:23:47.613626 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.609251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.0.98.2:37309)
0927 03:23:47.609701 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:47.609704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:47.609712 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:23:47.609748 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:47.609750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:47.609763 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:47.609978 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:47.610047 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.611208 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.612241 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.612990 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.613444 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.613509 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:47.613534 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 03:23:47.613567 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":71,"threads_started":1}
I20260927 03:23:47.613713 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.609451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.0.98.1:33605)
0927 03:23:47.609615 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:47.609620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:47.609629 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:23:47.609706 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:47.609708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:47.609725 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:47.609802 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:47.609894 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.611038 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.612322 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.612827 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.612905 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.612969 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:47.613590 (+ 621us) server_negotiation.cc:299] Negotiation successful
0927 03:23:47.613636 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260927 03:23:47.613909 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.609150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43655 (local address 127.0.0.1:47702)
0927 03:23:47.609439 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:47.609451 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:47.609499 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:47.609705 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:47.609708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:47.609710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:47.609775 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.609777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.610785 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.610786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.612143 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.612146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.613074 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.613075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.613141 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.613143 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:23:47.613191 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:23:47.613321 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20260927 03:23:47.613909 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.609147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37309 (local address 127.0.0.1:55978)
0927 03:23:47.609350 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:47.609392 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:47.609459 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:47.609851 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:47.609851 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:47.609854 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:47.609909 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.609910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.611327 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.611328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.612159 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.612161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.613162 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.613163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.613282 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.613286 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:23:47.613319 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 03:23:47.613362 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":3,"thread_start_us":127,"threads_started":1}
I20260927 03:23:47.614439 10746 tablet_service.cc:1467] Processing CreateTablet for tablet b17f833ca11a47429e01fd1170d40dd7 (DEFAULT_TABLE table=test-table [id=ca270f17bac94cd69881dd72b23ebc54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:47.614805 10746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17f833ca11a47429e01fd1170d40dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:47.614746 10880 tablet_service.cc:1467] Processing CreateTablet for tablet b17f833ca11a47429e01fd1170d40dd7 (DEFAULT_TABLE table=test-table [id=ca270f17bac94cd69881dd72b23ebc54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:47.615070 10880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17f833ca11a47429e01fd1170d40dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:47.617398 11094 tablet_bootstrap.cc:492] T b17f833ca11a47429e01fd1170d40dd7 P 6693b463eb0b43f2bec39cc0c4a5917e: Bootstrap starting.
I20260927 03:23:47.617856 11014 tablet_service.cc:1467] Processing CreateTablet for tablet b17f833ca11a47429e01fd1170d40dd7 (DEFAULT_TABLE table=test-table [id=ca270f17bac94cd69881dd72b23ebc54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:23:47.618144 11094 tablet_bootstrap.cc:654] T b17f833ca11a47429e01fd1170d40dd7 P 6693b463eb0b43f2bec39cc0c4a5917e: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:47.618150 11014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17f833ca11a47429e01fd1170d40dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:23:47.618614 11094 log.cc:824] T b17f833ca11a47429e01fd1170d40dd7 P 6693b463eb0b43f2bec39cc0c4a5917e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:47.619076 11095 tablet_bootstrap.cc:492] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5: Bootstrap starting.
I20260927 03:23:47.619405 11094 tablet_bootstrap.cc:492] T b17f833ca11a47429e01fd1170d40dd7 P 6693b463eb0b43f2bec39cc0c4a5917e: No bootstrap required, opened a new log
I20260927 03:23:47.619480 11094 ts_tablet_manager.cc:1397] T b17f833ca11a47429e01fd1170d40dd7 P 6693b463eb0b43f2bec39cc0c4a5917e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:23:47.619729 11095 tablet_bootstrap.cc:654] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:47.620270 11095 log.cc:824] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:47.620918 11098 tablet_bootstrap.cc:492] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba: Bootstrap starting.
I20260927 03:23:47.621394 11095 tablet_bootstrap.cc:492] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5: No bootstrap required, opened a new log
I20260927 03:23:47.621459 11095 ts_tablet_manager.cc:1397] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:23:47.621556 11098 tablet_bootstrap.cc:654] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba: Neither blocks nor log segments found. Creating new log.
I20260927 03:23:47.621419 11094 raft_consensus.cc:348] T b17f833ca11a47429e01fd1170d40dd7 P 6693b463eb0b43f2bec39cc0c4a5917e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa044013a014946b6d41bdd10f65fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43655 } } peers { permanent_uuid: "32e8ae455b20488bb8640f934c46bad5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "6693b463eb0b43f2bec39cc0c4a5917e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260927 03:23:47.621604 11094 raft_consensus.cc:374] T b17f833ca11a47429e01fd1170d40dd7 P 6693b463eb0b43f2bec39cc0c4a5917e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:47.621645 11094 raft_consensus.cc:729] T b17f833ca11a47429e01fd1170d40dd7 P 6693b463eb0b43f2bec39cc0c4a5917e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6693b463eb0b43f2bec39cc0c4a5917e, State: Initialized, Role: FOLLOWER
I20260927 03:23:47.621773 11094 consensus_queue.cc:260] T b17f833ca11a47429e01fd1170d40dd7 P 6693b463eb0b43f2bec39cc0c4a5917e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa044013a014946b6d41bdd10f65fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43655 } } peers { permanent_uuid: "32e8ae455b20488bb8640f934c46bad5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "6693b463eb0b43f2bec39cc0c4a5917e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260927 03:23:47.622081 11094 ts_tablet_manager.cc:1428] T b17f833ca11a47429e01fd1170d40dd7 P 6693b463eb0b43f2bec39cc0c4a5917e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:23:47.622102 11098 log.cc:824] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba: Log is configured to *not* fsync() on all Append() calls
I20260927 03:23:47.622735 11098 tablet_bootstrap.cc:492] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba: No bootstrap required, opened a new log
I20260927 03:23:47.622787 11098 ts_tablet_manager.cc:1397] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:23:47.623219 11095 raft_consensus.cc:348] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa044013a014946b6d41bdd10f65fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43655 } } peers { permanent_uuid: "32e8ae455b20488bb8640f934c46bad5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "6693b463eb0b43f2bec39cc0c4a5917e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260927 03:23:47.623387 11095 raft_consensus.cc:374] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:47.623422 11095 raft_consensus.cc:729] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e8ae455b20488bb8640f934c46bad5, State: Initialized, Role: FOLLOWER
I20260927 03:23:47.623519 11095 consensus_queue.cc:260] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa044013a014946b6d41bdd10f65fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43655 } } peers { permanent_uuid: "32e8ae455b20488bb8640f934c46bad5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "6693b463eb0b43f2bec39cc0c4a5917e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260927 03:23:47.623786 11095 ts_tablet_manager.cc:1428] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:23:47.623864 10946 heartbeater.cc:503] Master 127.0.98.62:39419 was elected leader, sending a full tablet report...
I20260927 03:23:47.623886 10812 heartbeater.cc:503] Master 127.0.98.62:39419 was elected leader, sending a full tablet report...
I20260927 03:23:47.624395 11098 raft_consensus.cc:348] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa044013a014946b6d41bdd10f65fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43655 } } peers { permanent_uuid: "32e8ae455b20488bb8640f934c46bad5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "6693b463eb0b43f2bec39cc0c4a5917e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260927 03:23:47.624559 11098 raft_consensus.cc:374] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:23:47.624593 11098 raft_consensus.cc:729] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa044013a014946b6d41bdd10f65fba, State: Initialized, Role: FOLLOWER
I20260927 03:23:47.624722 11098 consensus_queue.cc:260] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa044013a014946b6d41bdd10f65fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43655 } } peers { permanent_uuid: "32e8ae455b20488bb8640f934c46bad5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "6693b463eb0b43f2bec39cc0c4a5917e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260927 03:23:47.624976 11098 ts_tablet_manager.cc:1428] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:23:47.624995 11080 heartbeater.cc:503] Master 127.0.98.62:39419 was elected leader, sending a full tablet report...
W20260927 03:23:47.666769 10947 tablet.cc:2369] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:23:47.772492 10813 tablet.cc:2369] T b17f833ca11a47429e01fd1170d40dd7 P 6693b463eb0b43f2bec39cc0c4a5917e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:23:47.836936 11081 tablet.cc:2369] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:23:47.873049 11101 raft_consensus.cc:482] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:23:47.873175 11101 raft_consensus.cc:504] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa044013a014946b6d41bdd10f65fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43655 } } peers { permanent_uuid: "32e8ae455b20488bb8640f934c46bad5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "6693b463eb0b43f2bec39cc0c4a5917e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260927 03:23:47.873528 11101 leader_election.cc:288] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers caa044013a014946b6d41bdd10f65fba (127.0.98.3:43655), 6693b463eb0b43f2bec39cc0c4a5917e (127.0.98.1:33605)
I20260927 03:23:47.876317 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.873803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48249 (local address 127.0.98.1:33605)
0927 03:23:47.873846 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:47.873848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:47.873867 (+ 19us) server_negotiation.cc:407] Connection header received
0927 03:23:47.873904 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:47.873905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:47.873913 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:47.873952 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:47.874106 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.874641 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.875660 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.876005 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.876064 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.876076 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:47.876218 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 03:23:47.876253 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:23:47.876348 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.873718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33605 (local address 127.0.98.2:48249)
0927 03:23:47.873762 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:47.873776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:47.873823 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:47.873963 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:47.873964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:47.873966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:47.874041 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.874042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.874749 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.874750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.875570 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.875572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.876074 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.876075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.876150 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.876153 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:47.876197 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:23:47.876240 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":31}
I20260927 03:23:47.876338 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.873819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56149 (local address 127.0.98.3:43655)
0927 03:23:47.873877 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:47.873879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:47.873953 (+ 74us) server_negotiation.cc:407] Connection header received
0927 03:23:47.874006 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:47.874006 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:47.874017 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:47.874058 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:23:47.874200 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.874676 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.875680 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.876035 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.876092 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.876102 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:47.876235 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 03:23:47.876269 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:23:47.876365 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.873728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43655 (local address 127.0.98.2:56149)
0927 03:23:47.873879 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:47.873887 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:23:47.873948 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:23:47.874074 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:23:47.874074 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:23:47.874076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:23:47.874139 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.874140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.874762 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.874763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.875594 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:23:47.875596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.876105 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.876105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:23:47.876172 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.876175 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:23:47.876220 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:23:47.876257 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":37,"thread_start_us":86,"threads_started":1}
I20260927 03:23:47.876641 11034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17f833ca11a47429e01fd1170d40dd7" candidate_uuid: "32e8ae455b20488bb8640f934c46bad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa044013a014946b6d41bdd10f65fba" is_pre_election: true
I20260927 03:23:47.876685 10766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17f833ca11a47429e01fd1170d40dd7" candidate_uuid: "32e8ae455b20488bb8640f934c46bad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6693b463eb0b43f2bec39cc0c4a5917e" is_pre_election: true
I20260927 03:23:47.876793 11034 raft_consensus.cc:2454] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e8ae455b20488bb8640f934c46bad5 in term 0.
I20260927 03:23:47.876842 10766 raft_consensus.cc:2454] T b17f833ca11a47429e01fd1170d40dd7 P 6693b463eb0b43f2bec39cc0c4a5917e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e8ae455b20488bb8640f934c46bad5 in term 0.
I20260927 03:23:47.877039 10833 leader_election.cc:302] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e8ae455b20488bb8640f934c46bad5, caa044013a014946b6d41bdd10f65fba; no voters:
I20260927 03:23:47.877213 11101 raft_consensus.cc:2781] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:23:47.877262 11101 raft_consensus.cc:482] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:23:47.877285 11101 raft_consensus.cc:3037] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:47.878002 11101 raft_consensus.cc:504] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa044013a014946b6d41bdd10f65fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43655 } } peers { permanent_uuid: "32e8ae455b20488bb8640f934c46bad5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "6693b463eb0b43f2bec39cc0c4a5917e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260927 03:23:47.878126 11101 leader_election.cc:288] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [CANDIDATE]: Term 1 election: Requested vote from peers caa044013a014946b6d41bdd10f65fba (127.0.98.3:43655), 6693b463eb0b43f2bec39cc0c4a5917e (127.0.98.1:33605)
I20260927 03:23:47.878289 11034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17f833ca11a47429e01fd1170d40dd7" candidate_uuid: "32e8ae455b20488bb8640f934c46bad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa044013a014946b6d41bdd10f65fba"
I20260927 03:23:47.878328 10766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17f833ca11a47429e01fd1170d40dd7" candidate_uuid: "32e8ae455b20488bb8640f934c46bad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6693b463eb0b43f2bec39cc0c4a5917e"
I20260927 03:23:47.878388 10766 raft_consensus.cc:3037] T b17f833ca11a47429e01fd1170d40dd7 P 6693b463eb0b43f2bec39cc0c4a5917e [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:47.878389 11034 raft_consensus.cc:3037] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:23:47.879251 11034 raft_consensus.cc:2454] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e8ae455b20488bb8640f934c46bad5 in term 1.
I20260927 03:23:47.879251 10766 raft_consensus.cc:2454] T b17f833ca11a47429e01fd1170d40dd7 P 6693b463eb0b43f2bec39cc0c4a5917e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e8ae455b20488bb8640f934c46bad5 in term 1.
I20260927 03:23:47.879432 10836 leader_election.cc:302] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e8ae455b20488bb8640f934c46bad5, 6693b463eb0b43f2bec39cc0c4a5917e; no voters:
I20260927 03:23:47.879506 11101 raft_consensus.cc:2781] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:23:47.879628 11101 raft_consensus.cc:686] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [term 1 LEADER]: Becoming Leader. State: Replica: 32e8ae455b20488bb8640f934c46bad5, State: Running, Role: LEADER
I20260927 03:23:47.879707 11101 consensus_queue.cc:237] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa044013a014946b6d41bdd10f65fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43655 } } peers { permanent_uuid: "32e8ae455b20488bb8640f934c46bad5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "6693b463eb0b43f2bec39cc0c4a5917e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } }
I20260927 03:23:47.880599 10620 catalog_manager.cc:5495] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32e8ae455b20488bb8640f934c46bad5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "32e8ae455b20488bb8640f934c46bad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa044013a014946b6d41bdd10f65fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e8ae455b20488bb8640f934c46bad5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6693b463eb0b43f2bec39cc0c4a5917e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 } health_report { overall_health: UNKNOWN } } }
I20260927 03:23:47.896104 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.893676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55982 (local address 127.0.98.2:37309)
0927 03:23:47.893727 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:23:47.893729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:23:47.893738 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:23:47.893772 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:23:47.893773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:23:47.893792 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:23:47.893859 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:23:47.894054 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.894547 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.895435 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:23:47.895628 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:23:47.895696 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:23:47.895829 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:23:47.895938 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:23:47.895999 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:23:47.896013 (+ 14us) server_negotiation.cc:299] Negotiation successful
0927 03:23:47.896030 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:23:47.897914 10766 raft_consensus.cc:1261] T b17f833ca11a47429e01fd1170d40dd7 P 6693b463eb0b43f2bec39cc0c4a5917e [term 1 FOLLOWER]: Refusing update from remote peer 32e8ae455b20488bb8640f934c46bad5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:47.897964 11034 raft_consensus.cc:1261] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba [term 1 FOLLOWER]: Refusing update from remote peer 32e8ae455b20488bb8640f934c46bad5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:23:47.898325 11104 consensus_queue.cc:1035] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "caa044013a014946b6d41bdd10f65fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:47.898459 11101 consensus_queue.cc:1035] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6693b463eb0b43f2bec39cc0c4a5917e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:23:47.900772 11107 mvcc.cc:205] Tried to move back new op lower bound from 7333803736666963968 to 7333803736595914752. Current Snapshot: MvccSnapshot[applied={T|T < 7333803736666963968}]
I20260927 03:23:47.903610 10620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:47.903723 10620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:23:47.904410 10620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3326cf3e8ab7448ca024286ceec4cd0d: Sending DeleteTablet for 3 replicas of tablet b17f833ca11a47429e01fd1170d40dd7
I20260927 03:23:47.904758 10880 tablet_service.cc:1514] Processing DeleteTablet for tablet b17f833ca11a47429e01fd1170d40dd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55978
I20260927 03:23:47.904778 11014 tablet_service.cc:1514] Processing DeleteTablet for tablet b17f833ca11a47429e01fd1170d40dd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47702
I20260927 03:23:47.905011 11118 tablet_replica.cc:330] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba: stopping tablet replica
I20260927 03:23:47.905009 10746 tablet_service.cc:1514] Processing DeleteTablet for tablet b17f833ca11a47429e01fd1170d40dd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:23:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52166
I20260927 03:23:47.905187 11117 tablet_replica.cc:330] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5: stopping tablet replica
I20260927 03:23:47.905277 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 10683
I20260927 03:23:47.905316 11117 raft_consensus.cc:2229] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:23:47.905431 11117 raft_consensus.cc:2258] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:47.906263 11117 ts_tablet_manager.cc:1905] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:47.907998 11117 ts_tablet_manager.cc:1918] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:47.908056 11117 log.cc:1196] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/b17f833ca11a47429e01fd1170d40dd7
I20260927 03:23:47.908273 11117 ts_tablet_manager.cc:1939] T b17f833ca11a47429e01fd1170d40dd7 P 32e8ae455b20488bb8640f934c46bad5: Deleting consensus metadata
I20260927 03:23:47.908720 10607 catalog_manager.cc:4841] TS 32e8ae455b20488bb8640f934c46bad5 (127.0.98.2:37309): tablet b17f833ca11a47429e01fd1170d40dd7 (table test-table [id=ca270f17bac94cd69881dd72b23ebc54]) successfully deleted
I20260927 03:23:47.908936 11118 raft_consensus.cc:2229] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:23:47.909056 11118 raft_consensus.cc:2258] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:23:47.909463 11118 ts_tablet_manager.cc:1905] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:23:47.910678 11118 ts_tablet_manager.cc:1918] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:23:47.910748 11118 log.cc:1196] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/b17f833ca11a47429e01fd1170d40dd7
I20260927 03:23:47.911020 11118 ts_tablet_manager.cc:1939] T b17f833ca11a47429e01fd1170d40dd7 P caa044013a014946b6d41bdd10f65fba: Deleting consensus metadata
I20260927 03:23:47.911398 10605 catalog_manager.cc:4841] TS caa044013a014946b6d41bdd10f65fba (127.0.98.3:43655): tablet b17f833ca11a47429e01fd1170d40dd7 (table test-table [id=ca270f17bac94cd69881dd72b23ebc54]) successfully deleted
W20260927 03:23:47.913424 10608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33605 (error 108)
I20260927 03:23:47.913905 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:23:47.913778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33605 (local address 127.0.0.1:52172)
0927 03:23:47.913829 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:23:47.913869 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 03:23:47.913923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 10817
W20260927 03:23:47.915053 10608 catalog_manager.cc:4584] TS 6693b463eb0b43f2bec39cc0c4a5917e (127.0.98.1:33605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b17f833ca11a47429e01fd1170d40dd7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33605: connect: Connection refused (error 111)
I20260927 03:23:47.918237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 10951
I20260927 03:23:47.922257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 10589
2026-09-27T03:23:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2039 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:23:47 dist-test-slave-gw2v krb5kdc[11123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:23:47 dist-test-slave-gw2v krb5kdc[11123](info): set up 2 sockets
Sep 27 03:23:47 dist-test-slave-gw2v krb5kdc[11123](info): commencing operation
krb5kdc: starting...
W20260927 03:23:48.875552 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:23:48 dist-test-slave-gw2v krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:23:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:23:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 03:23:49.038570 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskzllPpO/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/postgres -l logfile start
2026-09-27 03:23:51.563 UTC [11153] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 03:23:51.563 UTC [11153] LOG: listening on IPv4 address "127.0.98.20", port 53147
2026-09-27 03:23:51.567 UTC [11153] LOG: listening on Unix socket "/tmp/.s.PGSQL.53147"
2026-09-27 03:23:51.595 UTC [11156] LOG: database system was shut down at 2026-09-27 03:23:50 UTC
2026-09-27 03:23:51.599 UTC [11153] LOG: database system is ready to accept connections
I20260927 03:23:52.467310 392 mini_postgres.cc:96] Postgres bound to 53147
2026-09-27 03:23:52.473 UTC [11164] FATAL: database "slave" does not exist
127.0.98.20:53147 - accepting connections
I20260927 03:23:52.473618 392 mini_ranger.cc:161] Starting Ranger...
I20260927 03:23:52.487891 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 03:23:52.986393 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 03:23:52.986474 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-admin
2026-09-27 03:23:53,249 [I] DB FLAVOR :POSTGRES
2026-09-27 03:23:53,250 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:23:53,250 [I] Checking connection..
2026-09-27 03:23:53,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 03:23:53,513 [I] Checking connection passed.
2026-09-27 03:23:53,513 [I] --------- Verifying version history table ---------
2026-09-27 03:23:53,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:23:53,772 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:23:53,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:23:54,033 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:23:54,033 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 03:23:54,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 03:23:54.283 UTC [11228] WARNING: there is no transaction in progress
2026-09-27 03:23:54,289 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 03:23:54,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:23:54,514 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 03:23:54,514 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:23:54,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 03:23:54,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 03:23:54,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gw2v.c.gcp-upstream.internal','N') ;"
2026-09-27 03:23:55,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:23:55,478 [I] Table x_portal_user does not exist in database ranger
2026-09-27 03:23:55,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:23:55,723 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 03:23:55,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:23:55,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:23:56,240 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 03:23:56,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 03:23:57.240 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:57.256 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:57.273 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:57.289 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:57.305 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:57.323 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:57.379 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:57.386 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:57.401 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:57.411 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:57.422 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:57.436 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:57.446 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:57.452 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:57.459 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:57.466 UTC [11336] WARNING: there is no transaction in progress
2026-09-27 03:23:58,156 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 03:23:58,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 03:23:58,398 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 03:23:58,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:23:58,674 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 03:23:58,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:23:58,905 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 03:23:58,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:23:59,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:23:59,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gw2v.c.gcp-upstream.internal';"
2026-09-27 03:23:59,648 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 03:23:59,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 03:23:59,893 [I] DB_PATCHES have already been applied
I20260927 03:23:59.898756 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskzllPpO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 03:23:59.898861 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 03:23:59.901746 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:46307
Sep 27, 2026 3:24:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:24:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:24:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:24:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:24:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:24:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:24:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 03:24:00.906287 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.433s user 0.002s sys 0.007s
Sep 27, 2026 3:24:20 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:24:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 3:24:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:24:20 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:24:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:24:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:24:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:24:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:24:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:24:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:24:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:24:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:24:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:24:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:24:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:24:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:24:20 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:24:21 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 03:24:22.279361 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 03:24:22 dist-test-slave-gw2v krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479462, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 03:24:22.373411 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 03:24:22.382602 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 03:24:23.239198 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 03:24:23.239279 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-kms
I20260927 03:24:23.239295 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:53147
2026-09-27 03:24:23,400 [I] DB FLAVOR :POSTGRES
2026-09-27 03:24:23,400 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:24:23,400 [I] Checking connection
2026-09-27 03:24:23,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 03:24:23,649 [I] connection success
2026-09-27 03:24:23,649 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 03:24:23,649 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 03:24:23,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 03:24:23,901 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 03:24:23,901 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:24:23,901 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 03:24:23,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53147/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 03:24:24,158 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 03:24:24.630451 392 mini_ranger_kms.cc:310] Created kms service
I20260927 03:24:24.832113 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 03:24:24.873535 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 03:24:24 dist-test-slave-gw2v krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479462, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:24:25.278312 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 03:24:25.278429 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 03:24:25.278448 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 03:24:25.281947 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11516
I20260927 03:24:25.282089 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:47491
Sep 27, 2026 3:24:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:24:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:24:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:24:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:24:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:24:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 03:24:27.322464 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.949s user 0.006s sys 0.138s
I20260927 03:24:27.322571 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 03:24:27.322638 392 mini_ranger_kms.cc:401] 127.0.98.20:47491/kms/v1/keys
Sep 27 03:24:27 dist-test-slave-gw2v krb5kdc[11123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790479467, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 03:24:27 dist-test-slave-gw2v krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479467, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 03:24:28 dist-test-slave-gw2v krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479467, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 03:24:30 dist-test-slave-gw2v krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:24:30.422297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40863
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40863
--ranger_config_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47491/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:24:30.502686 11567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:24:30.502928 11567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:24:30.502959 11567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:24:30.504136 11567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:24:30.504185 11567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:24:30.504202 11567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:24:30.504217 11567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:24:30.504233 11567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:24:30.505780 11567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:47491/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40863
--ranger_config_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:24:30.506028 11567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:24:30.506243 11567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:24:30.511590 11573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:24:30.511613 11575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:24:30.511724 11567 server_base.cc:1034] running on GCE node
W20260927 03:24:30.511631 11572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:24:30.512048 11567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:24:30.512372 11567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:24:30.513530 11567 hybrid_clock.cc:648] HybridClock initialized: now 1790479470513507 us; error 38 us; skew 500 ppm
Sep 27 03:24:30 dist-test-slave-gw2v krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:24:30.516067 11567 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:24:30.516497 11567 webserver.cc:460] Webserver started at http://127.0.98.62:41843/ using document root <none> and password file <none>
I20260927 03:24:30.516671 11567 fs_manager.cc:359] Metadata directory not provided
I20260927 03:24:30.516705 11567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:24:30.516800 11567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:24:30 dist-test-slave-gw2v krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:24:30.570574 11567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "5b62b8179b1841509a2e748b453c0a9f"
format_stamp: "Formatted at 2026-09-27 03:24:30 on dist-test-slave-gw2v"
server_key: "57968a03d69370b9a0b1a79951ef8b14"
server_key_iv: "91f8cf2f036a84242d96feb8b24a7dc1"
server_key_version: "kuduclusterkey@0"
I20260927 03:24:30.571085 11567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "5b62b8179b1841509a2e748b453c0a9f"
format_stamp: "Formatted at 2026-09-27 03:24:30 on dist-test-slave-gw2v"
server_key: "57968a03d69370b9a0b1a79951ef8b14"
server_key_iv: "91f8cf2f036a84242d96feb8b24a7dc1"
server_key_version: "kuduclusterkey@0"
I20260927 03:24:30.572925 11567 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260927 03:24:30.588444 11582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:24:30.588671 11567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:24:30.588779 11567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "5b62b8179b1841509a2e748b453c0a9f"
format_stamp: "Formatted at 2026-09-27 03:24:30 on dist-test-slave-gw2v"
server_key: "57968a03d69370b9a0b1a79951ef8b14"
server_key_iv: "91f8cf2f036a84242d96feb8b24a7dc1"
server_key_version: "kuduclusterkey@0"
I20260927 03:24:30.588888 11567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:24:30.617723 11567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:24:30.620002 11567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:24:30.620167 11567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:24:30.625123 11567 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40863
I20260927 03:24:30.625145 11634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40863 every 8 connection(s)
I20260927 03:24:30.625571 11567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:24:30.626286 11635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:24:30.628814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 11567
I20260927 03:24:30.628916 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:24:30.629096 392 external_mini_cluster.cc:1428] Setting key 7dbca029fcb95a938a9b8db37bc5a13e
I20260927 03:24:30.629164 11635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f: Bootstrap starting.
I20260927 03:24:30.630059 11635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f: Neither blocks nor log segments found. Creating new log.
I20260927 03:24:30.630358 11635 log.cc:824] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:24:30.631047 11635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f: No bootstrap required, opened a new log
I20260927 03:24:30.633248 11635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b62b8179b1841509a2e748b453c0a9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40863 } }
I20260927 03:24:30.633388 11635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:24:30.633420 11635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b62b8179b1841509a2e748b453c0a9f, State: Initialized, Role: FOLLOWER
I20260927 03:24:30.633543 11635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b62b8179b1841509a2e748b453c0a9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40863 } }
I20260927 03:24:30.633617 11635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:24:30.633651 11635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:24:30.633687 11635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:24:30.634318 11635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b62b8179b1841509a2e748b453c0a9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40863 } }
I20260927 03:24:30.634409 11635 leader_election.cc:302] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b62b8179b1841509a2e748b453c0a9f; no voters:
I20260927 03:24:30.634557 11635 leader_election.cc:288] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:24:30.634703 11640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:24:30.634838 11635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:24:30.634886 11640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [term 1 LEADER]: Becoming Leader. State: Replica: 5b62b8179b1841509a2e748b453c0a9f, State: Running, Role: LEADER
I20260927 03:24:30.634995 11640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b62b8179b1841509a2e748b453c0a9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40863 } }
I20260927 03:24:30.635279 11640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b62b8179b1841509a2e748b453c0a9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b62b8179b1841509a2e748b453c0a9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40863 } } }
I20260927 03:24:30.635341 11640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [sys.catalog]: This master's current role is: LEADER
I20260927 03:24:30.635447 11640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b62b8179b1841509a2e748b453c0a9f. Latest consensus state: current_term: 1 leader_uuid: "5b62b8179b1841509a2e748b453c0a9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b62b8179b1841509a2e748b453c0a9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40863 } } }
I20260927 03:24:30.635494 11640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f [sys.catalog]: This master's current role is: LEADER
I20260927 03:24:30.635653 11644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:24:30.636153 11644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:24:30 dist-test-slave-gw2v krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:24:30.637089 11635 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 03:24:30.642488 11644 catalog_manager.cc:1347] Generated new cluster ID: 825688ca17344bbd94d494d0a3381dc0
I20260927 03:24:30.642614 11644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:24:30.644757 11638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:30.629597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59098 (local address 127.0.98.62:40863)
0927 03:24:30.629766 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 03:24:30.629777 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:24:30.629809 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:24:30.630357 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:24:30.630367 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:24:30.630430 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:24:30.630583 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:24:30.631284 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:30.632646 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:30.633699 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:30.634333 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:30.634444 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:30.637148 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:24:30.637153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:24:30.637153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:24:30.637164 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:24:30.642680 (+ 5516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:24:30.643139 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:24:30.643143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:24:30.643143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:24:30.643250 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:24:30.643433 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:24:30.643434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:24:30.643434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:24:30.643749 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:24:30.643907 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 03:24:30.644385 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 03:24:30.644538 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260927 03:24:30.650923 11644 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:24:30.651671 11644 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:24:30.664669 11644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f: Generated new TSK 0
I20260927 03:24:30.664988 11644 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:24:31.365927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47491/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40863
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:24:31.450582 11680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:24:31.450783 11680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:24:31.450804 11680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:24:31.451958 11680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:24:31.451998 11680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:24:31.452023 11680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:24:31.453433 11680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:47491/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:24:31.453630 11680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:24:31.453853 11680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:24:31.459889 11685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:24:31.459924 11680 server_base.cc:1034] running on GCE node
W20260927 03:24:31.459906 11686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:24:31.460004 11688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:24:31.460430 11680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:24:31.460661 11680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:24:31.461791 11680 hybrid_clock.cc:648] HybridClock initialized: now 1790479471461780 us; error 30 us; skew 500 ppm
Sep 27 03:24:31 dist-test-slave-gw2v krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:24:31.464385 11680 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:24:31.464829 11680 webserver.cc:460] Webserver started at http://127.0.98.1:45649/ using document root <none> and password file <none>
I20260927 03:24:31.465034 11680 fs_manager.cc:359] Metadata directory not provided
I20260927 03:24:31.465085 11680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:24:31.465202 11680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:24:31 dist-test-slave-gw2v krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:24:31.476935 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "dfa29204d89744cc9528b20f75c8d21d"
format_stamp: "Formatted at 2026-09-27 03:24:31 on dist-test-slave-gw2v"
server_key: "621cb0f26fb157aaac59a922c2189672"
server_key_iv: "8acd4b1c66e5e798c02e30e42415d170"
server_key_version: "kuduclusterkey@0"
I20260927 03:24:31.477451 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dfa29204d89744cc9528b20f75c8d21d"
format_stamp: "Formatted at 2026-09-27 03:24:31 on dist-test-slave-gw2v"
server_key: "621cb0f26fb157aaac59a922c2189672"
server_key_iv: "8acd4b1c66e5e798c02e30e42415d170"
server_key_version: "kuduclusterkey@0"
I20260927 03:24:31.479112 11680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:24:31.491447 11695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:24:31.491645 11680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:24:31.491755 11680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "dfa29204d89744cc9528b20f75c8d21d"
format_stamp: "Formatted at 2026-09-27 03:24:31 on dist-test-slave-gw2v"
server_key: "621cb0f26fb157aaac59a922c2189672"
server_key_iv: "8acd4b1c66e5e798c02e30e42415d170"
server_key_version: "kuduclusterkey@0"
I20260927 03:24:31.491849 11680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:24:31.520196 11680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:24:31.522648 11680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:24:31.522888 11680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:24:31.523147 11680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:24:31.523517 11680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:24:31.523577 11680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:24:31.523622 11680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:24:31.523654 11680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:24:31.530161 11680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38517
I20260927 03:24:31.530290 11810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38517 every 8 connection(s)
I20260927 03:24:31.530575 11680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:24:31.531688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 11680
I20260927 03:24:31.531772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:24:31.531939 392 external_mini_cluster.cc:1428] Setting key 48369ad8459b7d8086738308e832bc58
Sep 27 03:24:31 dist-test-slave-gw2v krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:24:31.542538 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:31.531291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40863 (local address 127.0.98.1:57579)
0927 03:24:31.531467 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:24:31.531518 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 03:24:31.532489 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:24:31.532914 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:24:31.532919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:24:31.533168 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:24:31.533459 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:31.533466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:31.534484 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:31.534485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:31.535877 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:31.535880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:31.536451 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:31.536452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:31.536604 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:31.537254 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:24:31.537264 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:24:31.538921 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:24:31.540925 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:24:31.540934 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:24:31.540936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:24:31.541265 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:24:31.541506 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:24:31.541506 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:24:31.541507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:24:31.541633 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:24:31.541914 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:24:31.541920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:24:31.542104 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 03:24:31.542209 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 03:24:31.542363 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260927 03:24:31.543109 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:31.531360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57579 (local address 127.0.98.62:40863)
0927 03:24:31.532693 (+ 1333us) server_negotiation.cc:206] Beginning negotiation
0927 03:24:31.532695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:24:31.532702 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:24:31.532723 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:24:31.532724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:24:31.532736 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:24:31.532802 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:24:31.533568 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:31.534377 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:31.536046 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:31.536357 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:31.539015 (+ 2658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:31.539071 (+ 56us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:24:31.539072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:24:31.539072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:24:31.539076 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:24:31.540829 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:24:31.541379 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:24:31.541380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:24:31.541380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:24:31.541440 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:24:31.541698 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:24:31.541699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:24:31.541699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:24:31.541840 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:24:31.542805 (+ 965us) server_negotiation.cc:1035] Waiting for connection context
0927 03:24:31.542866 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 03:24:31.542946 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1303,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 03:24:31.543466 11811 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40863
I20260927 03:24:31.543568 11811 heartbeater.cc:464] Registering TS with master...
I20260927 03:24:31.543761 11811 heartbeater.cc:511] Master 127.0.98.62:40863 requested a full tablet report, sending...
I20260927 03:24:31.544317 11599 ts_manager.cc:194] Registered new tserver with Master: dfa29204d89744cc9528b20f75c8d21d (127.0.98.1:38517)
I20260927 03:24:31.544903 11599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:24:31.565510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47491/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40863
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:24:31.687140 11817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:24:31.687477 11817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:24:31.687548 11817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:24:31.689376 11817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:24:31.689435 11817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:24:31.689477 11817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:24:31.692142 11817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:47491/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:24:31.692427 11817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:24:31.692739 11817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:24:31.699106 11827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:24:31.699136 11824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:24:31.699381 11817 server_base.cc:1034] running on GCE node
W20260927 03:24:31.699182 11825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:24:31.699697 11817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:24:31.699985 11817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:24:31.701152 11817 hybrid_clock.cc:648] HybridClock initialized: now 1790479471701129 us; error 37 us; skew 500 ppm
Sep 27 03:24:31 dist-test-slave-gw2v krb5kdc[11123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790479471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 03:24:31 dist-test-slave-gw2v krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:24:31.708249 11817 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:24:31.708765 11817 webserver.cc:460] Webserver started at http://127.0.98.2:40675/ using document root <none> and password file <none>
I20260927 03:24:31.709040 11817 fs_manager.cc:359] Metadata directory not provided
I20260927 03:24:31.709107 11817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:24:31.709252 11817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:24:31 dist-test-slave-gw2v krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:24:31.731688 11817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "7abf79ebadac489db14166e69b3581ab"
format_stamp: "Formatted at 2026-09-27 03:24:31 on dist-test-slave-gw2v"
server_key: "c73744b325fdf292289bbbde0a9f5566"
server_key_iv: "29c2ef77e097e2320935fa9b6b22040b"
server_key_version: "kuduclusterkey@0"
I20260927 03:24:31.732203 11817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7abf79ebadac489db14166e69b3581ab"
format_stamp: "Formatted at 2026-09-27 03:24:31 on dist-test-slave-gw2v"
server_key: "c73744b325fdf292289bbbde0a9f5566"
server_key_iv: "29c2ef77e097e2320935fa9b6b22040b"
server_key_version: "kuduclusterkey@0"
I20260927 03:24:31.733999 11817 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 03:24:31.751363 11834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:24:31.751621 11817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:24:31.751717 11817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "7abf79ebadac489db14166e69b3581ab"
format_stamp: "Formatted at 2026-09-27 03:24:31 on dist-test-slave-gw2v"
server_key: "c73744b325fdf292289bbbde0a9f5566"
server_key_iv: "29c2ef77e097e2320935fa9b6b22040b"
server_key_version: "kuduclusterkey@0"
I20260927 03:24:31.751806 11817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:24:31.768278 11817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:24:31.770164 11817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:24:31.770293 11817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:24:31.770506 11817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:24:31.770876 11817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:24:31.770917 11817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:24:31.770948 11817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:24:31.770998 11817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:24:31.777647 11817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38137
I20260927 03:24:31.778156 11817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:24:31.779973 11947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38137 every 8 connection(s)
I20260927 03:24:31.783147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 11817
I20260927 03:24:31.783250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:24:31.783430 392 external_mini_cluster.cc:1428] Setting key ed1d6e990fd7d8b802b191f420b57f4c
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 03:24:31 dist-test-slave-gw2v krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:24:31.801234 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:31.780974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36051 (local address 127.0.98.62:40863)
0927 03:24:31.781052 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 03:24:31.781054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:24:31.791138 (+ 10084us) server_negotiation.cc:407] Connection header received
0927 03:24:31.791914 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:24:31.791917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:24:31.791930 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:24:31.791962 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:24:31.792857 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:31.793601 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:31.795149 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:31.795449 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:31.795495 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:31.798060 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:24:31.798061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:24:31.798062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:24:31.798069 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:24:31.799718 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:24:31.800172 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:24:31.800173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:24:31.800173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:24:31.800227 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:24:31.800482 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:24:31.800483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:24:31.800483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:24:31.800611 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:24:31.800666 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:24:31.801047 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 03:24:31.801133 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 03:24:31.801754 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:31.780683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40863 (local address 127.0.98.2:36051)
0927 03:24:31.790939 (+ 10256us) negotiation.cc:107] Waiting for socket to connect
0927 03:24:31.791014 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 03:24:31.791773 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:24:31.792177 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:24:31.792182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:24:31.792427 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:24:31.792744 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:31.792751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:31.793725 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:31.793727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:31.795049 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:31.795052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:31.795590 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:31.795591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:31.795745 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:31.796421 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:24:31.796432 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:24:31.797956 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:24:31.799808 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:24:31.799814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:24:31.799816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:24:31.800086 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:24:31.800290 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:24:31.800291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:24:31.800291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:24:31.800415 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:24:31.800711 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:24:31.800718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:24:31.800895 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 03:24:31.801358 (+ 463us) client_negotiation.cc:240] Negotiation successful
0927 03:24:31.801515 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10190,"thread_start_us":91,"threads_started":1}
I20260927 03:24:31.802170 11948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40863
I20260927 03:24:31.802258 11948 heartbeater.cc:464] Registering TS with master...
I20260927 03:24:31.802436 11948 heartbeater.cc:511] Master 127.0.98.62:40863 requested a full tablet report, sending...
I20260927 03:24:31.802783 11599 ts_manager.cc:194] Registered new tserver with Master: 7abf79ebadac489db14166e69b3581ab (127.0.98.2:38137)
I20260927 03:24:31.803270 11599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:24:31.825695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47491/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40863
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:24:31.954136 11954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:24:31.954517 11954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:24:31.954622 11954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:24:31.956795 11954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:24:31.956918 11954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:24:31.957051 11954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:24:31.959523 11954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:47491/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:24:31.960036 11954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:24:31.960390 11954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:24:31.968004 11960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:24:31.968415 11959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:24:31.968700 11962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:24:31.969772 11954 server_base.cc:1034] running on GCE node
I20260927 03:24:31.970151 11954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:24:31.970445 11954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:24:31.971586 11954 hybrid_clock.cc:648] HybridClock initialized: now 1790479471971562 us; error 37 us; skew 500 ppm
Sep 27 03:24:31 dist-test-slave-gw2v krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:24:31.975019 11954 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:24:31.975554 11954 webserver.cc:460] Webserver started at http://127.0.98.3:38199/ using document root <none> and password file <none>
I20260927 03:24:31.975835 11954 fs_manager.cc:359] Metadata directory not provided
I20260927 03:24:31.975888 11954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:24:31.976017 11954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:24:31 dist-test-slave-gw2v krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:24:31.993793 11954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "99bbf5ea4ffd485784b874b69c5967ac"
format_stamp: "Formatted at 2026-09-27 03:24:31 on dist-test-slave-gw2v"
server_key: "7c68a47934ad9d2365e396a31901b84d"
server_key_iv: "5d455889aee29100ab3212b1b7c00d19"
server_key_version: "kuduclusterkey@0"
I20260927 03:24:31.995119 11954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "99bbf5ea4ffd485784b874b69c5967ac"
format_stamp: "Formatted at 2026-09-27 03:24:31 on dist-test-slave-gw2v"
server_key: "7c68a47934ad9d2365e396a31901b84d"
server_key_iv: "5d455889aee29100ab3212b1b7c00d19"
server_key_version: "kuduclusterkey@0"
I20260927 03:24:31.996876 11954 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 03:24:32.011735 11969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:24:32.012006 11954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:24:32.012138 11954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "99bbf5ea4ffd485784b874b69c5967ac"
format_stamp: "Formatted at 2026-09-27 03:24:31 on dist-test-slave-gw2v"
server_key: "7c68a47934ad9d2365e396a31901b84d"
server_key_iv: "5d455889aee29100ab3212b1b7c00d19"
server_key_version: "kuduclusterkey@0"
I20260927 03:24:32.012270 11954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:24:32.046010 11954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:24:32.048655 11954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:24:32.048828 11954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:24:32.049095 11954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:24:32.049480 11954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:24:32.049544 11954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:24:32.049587 11954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:24:32.049609 11954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:24:32.057672 11954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34167
I20260927 03:24:32.058120 11954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:24:32.059654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 11954
I20260927 03:24:32.059751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:24:32.059949 392 external_mini_cluster.cc:1428] Setting key 56428e531e87b7094fc9bc89332b9267
I20260927 03:24:32.071242 12082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34167 every 8 connection(s)
Sep 27 03:24:32 dist-test-slave-gw2v krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:24:32.086679 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:32.071930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40035 (local address 127.0.98.62:40863)
0927 03:24:32.071990 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:24:32.071992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:24:32.072346 (+ 354us) server_negotiation.cc:407] Connection header received
0927 03:24:32.073072 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:24:32.073073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:24:32.073085 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:24:32.073113 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:24:32.074009 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.074768 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.076292 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.076608 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.076650 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:32.082564 (+ 5914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:24:32.082566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:24:32.082566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:24:32.082576 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:24:32.084308 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:24:32.084862 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:24:32.084863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:24:32.084864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:24:32.084927 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:24:32.085810 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:24:32.085811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:24:32.085812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:24:32.085970 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:24:32.086014 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:24:32.086465 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 03:24:32.086578 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:24:32.087266 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:32.071935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40863 (local address 127.0.98.3:40035)
0927 03:24:32.072219 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 03:24:32.072263 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:24:32.072943 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:24:32.073304 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:24:32.073308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:24:32.073618 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:24:32.073901 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:32.073907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.074902 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.074903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:32.076192 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:32.076195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.076743 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.076743 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:32.076892 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:32.077519 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:24:32.077529 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:24:32.082161 (+ 4632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:24:32.084400 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:24:32.084411 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:24:32.084415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:24:32.084763 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:24:32.085000 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:24:32.085001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:24:32.085001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:24:32.085166 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:24:32.086063 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:24:32.086071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:24:32.086282 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 03:24:32.086791 (+ 509us) client_negotiation.cc:240] Negotiation successful
0927 03:24:32.087015 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":207,"threads_started":1}
I20260927 03:24:32.087771 12083 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40863
I20260927 03:24:32.087878 12083 heartbeater.cc:464] Registering TS with master...
I20260927 03:24:32.088100 12083 heartbeater.cc:511] Master 127.0.98.62:40863 requested a full tablet report, sending...
I20260927 03:24:32.088470 11599 ts_manager.cc:194] Registered new tserver with Master: 99bbf5ea4ffd485784b874b69c5967ac (127.0.98.3:34167)
I20260927 03:24:32.089002 11599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40035
I20260927 03:24:32.093158 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:24:32.104738 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:32.096135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.0.98.62:40863)
0927 03:24:32.096181 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:24:32.096183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:24:32.096624 (+ 441us) server_negotiation.cc:407] Connection header received
0927 03:24:32.096735 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:24:32.096736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:24:32.096748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:24:32.096777 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:24:32.097572 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.098352 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.099499 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.099769 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.099848 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:32.101498 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:24:32.101499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:24:32.101500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:24:32.101515 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:24:32.103279 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:24:32.103694 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:24:32.103695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:24:32.103695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:24:32.103758 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:24:32.103956 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:24:32.103956 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:24:32.103957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:24:32.104089 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:24:32.104168 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:24:32.104380 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 03:24:32.104522 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:24:32.107082 11599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:24:32.109493 11599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:24:32.127410 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:32.113432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38517 (local address 127.0.0.1:36168)
0927 03:24:32.115185 (+ 1753us) negotiation.cc:107] Waiting for socket to connect
0927 03:24:32.115194 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:24:32.115236 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:24:32.115506 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:24:32.115510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:24:32.115512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:24:32.115577 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:32.115579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.117121 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.117122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:32.120473 (+ 3351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:32.120488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.127073 (+ 6585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.127074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:32.127172 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:32.127193 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 03:24:32.127264 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 03:24:32.127322 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1740,"thread_start_us":98,"threads_started":1}
I20260927 03:24:32.127768 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:32.113906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36168 (local address 127.0.98.1:38517)
0927 03:24:32.114468 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 03:24:32.114474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:24:32.115289 (+ 815us) server_negotiation.cc:407] Connection header received
0927 03:24:32.115327 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:24:32.115330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:24:32.115345 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:24:32.115412 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:24:32.115656 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.117034 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.120582 (+ 3548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.121662 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.121726 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:32.121786 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:24:32.127640 (+ 5854us) server_negotiation.cc:299] Negotiation successful
0927 03:24:32.127700 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":53,"threads_started":1}
I20260927 03:24:32.128286 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:32.113613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38137 (local address 127.0.0.1:58258)
0927 03:24:32.117585 (+ 3972us) negotiation.cc:107] Waiting for socket to connect
0927 03:24:32.117596 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:24:32.117651 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:24:32.117916 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:24:32.117917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:24:32.117919 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:24:32.117964 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:32.117966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.121816 (+ 3850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.121817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:32.122915 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:32.122917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.128052 (+ 5135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.128053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:32.128117 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:32.128122 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:24:32.128174 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:24:32.128215 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3956,"thread_start_us":50,"threads_started":1}
I20260927 03:24:32.128489 11745 tablet_service.cc:1467] Processing CreateTablet for tablet 3311ec07e0824495937a3161482d6e17 (DEFAULT_TABLE table=test-table [id=928fb8c8c87b4f8b987bd195dccf46a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:24:32.128881 11745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3311ec07e0824495937a3161482d6e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:24:32.128583 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:32.114035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58258 (local address 127.0.98.2:38137)
0927 03:24:32.114565 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0927 03:24:32.114568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:24:32.117707 (+ 3139us) server_negotiation.cc:407] Connection header received
0927 03:24:32.117752 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:24:32.117755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:24:32.117770 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:24:32.117836 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:24:32.118036 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.119444 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.123028 (+ 3584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.123813 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.123884 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:32.123959 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:24:32.128433 (+ 4474us) server_negotiation.cc:299] Negotiation successful
0927 03:24:32.128500 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":72,"threads_started":1}
I20260927 03:24:32.129565 11881 tablet_service.cc:1467] Processing CreateTablet for tablet 3311ec07e0824495937a3161482d6e17 (DEFAULT_TABLE table=test-table [id=928fb8c8c87b4f8b987bd195dccf46a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:24:32.129899 11881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3311ec07e0824495937a3161482d6e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:24:32.134796 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:32.112923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34167 (local address 127.0.0.1:59226)
0927 03:24:32.114954 (+ 2031us) negotiation.cc:107] Waiting for socket to connect
0927 03:24:32.114999 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:24:32.115066 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:24:32.124287 (+ 9221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:24:32.124288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:24:32.124291 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:24:32.124332 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:32.124333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.125871 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.125872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:32.126980 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:32.126983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.134532 (+ 7549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.134534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:32.134608 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:32.134613 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:24:32.134664 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:24:32.134710 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1982,"thread_start_us":79,"threads_started":1}
I20260927 03:24:32.135157 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:32.113225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59226 (local address 127.0.98.3:34167)
0927 03:24:32.124059 (+ 10834us) server_negotiation.cc:206] Beginning negotiation
0927 03:24:32.124064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:24:32.124074 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:24:32.124113 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:24:32.124115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:24:32.124131 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:24:32.124205 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:24:32.124411 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.125774 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.133334 (+ 7560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.134334 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.134407 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:32.134493 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:24:32.134994 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 03:24:32.135042 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10790,"thread_start_us":71,"threads_started":1}
I20260927 03:24:32.135802 12017 tablet_service.cc:1467] Processing CreateTablet for tablet 3311ec07e0824495937a3161482d6e17 (DEFAULT_TABLE table=test-table [id=928fb8c8c87b4f8b987bd195dccf46a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:24:32.136147 12017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3311ec07e0824495937a3161482d6e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:24:32.138994 12103 tablet_bootstrap.cc:492] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab: Bootstrap starting.
I20260927 03:24:32.139997 12103 tablet_bootstrap.cc:654] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab: Neither blocks nor log segments found. Creating new log.
I20260927 03:24:32.140370 12103 log.cc:824] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab: Log is configured to *not* fsync() on all Append() calls
I20260927 03:24:32.143460 12102 tablet_bootstrap.cc:492] T 3311ec07e0824495937a3161482d6e17 P dfa29204d89744cc9528b20f75c8d21d: Bootstrap starting.
I20260927 03:24:32.144688 12102 tablet_bootstrap.cc:654] T 3311ec07e0824495937a3161482d6e17 P dfa29204d89744cc9528b20f75c8d21d: Neither blocks nor log segments found. Creating new log.
I20260927 03:24:32.144968 12102 log.cc:824] T 3311ec07e0824495937a3161482d6e17 P dfa29204d89744cc9528b20f75c8d21d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:24:32.145803 12102 tablet_bootstrap.cc:492] T 3311ec07e0824495937a3161482d6e17 P dfa29204d89744cc9528b20f75c8d21d: No bootstrap required, opened a new log
I20260927 03:24:32.145922 12102 ts_tablet_manager.cc:1397] T 3311ec07e0824495937a3161482d6e17 P dfa29204d89744cc9528b20f75c8d21d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:24:32.147022 12104 tablet_bootstrap.cc:492] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac: Bootstrap starting.
I20260927 03:24:32.147418 12103 tablet_bootstrap.cc:492] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab: No bootstrap required, opened a new log
I20260927 03:24:32.147502 12103 ts_tablet_manager.cc:1397] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260927 03:24:32.148022 12104 tablet_bootstrap.cc:654] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac: Neither blocks nor log segments found. Creating new log.
I20260927 03:24:32.148331 12104 log.cc:824] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac: Log is configured to *not* fsync() on all Append() calls
I20260927 03:24:32.148904 12102 raft_consensus.cc:348] T 3311ec07e0824495937a3161482d6e17 P dfa29204d89744cc9528b20f75c8d21d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bbf5ea4ffd485784b874b69c5967ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34167 } } peers { permanent_uuid: "dfa29204d89744cc9528b20f75c8d21d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38517 } } peers { permanent_uuid: "7abf79ebadac489db14166e69b3581ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38137 } }
I20260927 03:24:32.149178 12102 raft_consensus.cc:374] T 3311ec07e0824495937a3161482d6e17 P dfa29204d89744cc9528b20f75c8d21d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:24:32.149277 12102 raft_consensus.cc:729] T 3311ec07e0824495937a3161482d6e17 P dfa29204d89744cc9528b20f75c8d21d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa29204d89744cc9528b20f75c8d21d, State: Initialized, Role: FOLLOWER
I20260927 03:24:32.149492 12102 consensus_queue.cc:260] T 3311ec07e0824495937a3161482d6e17 P dfa29204d89744cc9528b20f75c8d21d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bbf5ea4ffd485784b874b69c5967ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34167 } } peers { permanent_uuid: "dfa29204d89744cc9528b20f75c8d21d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38517 } } peers { permanent_uuid: "7abf79ebadac489db14166e69b3581ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38137 } }
I20260927 03:24:32.149510 12103 raft_consensus.cc:348] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bbf5ea4ffd485784b874b69c5967ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34167 } } peers { permanent_uuid: "dfa29204d89744cc9528b20f75c8d21d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38517 } } peers { permanent_uuid: "7abf79ebadac489db14166e69b3581ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38137 } }
I20260927 03:24:32.149670 12103 raft_consensus.cc:374] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:24:32.149710 12103 raft_consensus.cc:729] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abf79ebadac489db14166e69b3581ab, State: Initialized, Role: FOLLOWER
I20260927 03:24:32.149847 12103 consensus_queue.cc:260] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bbf5ea4ffd485784b874b69c5967ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34167 } } peers { permanent_uuid: "dfa29204d89744cc9528b20f75c8d21d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38517 } } peers { permanent_uuid: "7abf79ebadac489db14166e69b3581ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38137 } }
I20260927 03:24:32.150102 12103 ts_tablet_manager.cc:1428] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:24:32.150090 12102 ts_tablet_manager.cc:1428] T 3311ec07e0824495937a3161482d6e17 P dfa29204d89744cc9528b20f75c8d21d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 03:24:32.150133 11948 heartbeater.cc:503] Master 127.0.98.62:40863 was elected leader, sending a full tablet report...
I20260927 03:24:32.150297 11811 heartbeater.cc:503] Master 127.0.98.62:40863 was elected leader, sending a full tablet report...
I20260927 03:24:32.155457 12104 tablet_bootstrap.cc:492] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac: No bootstrap required, opened a new log
I20260927 03:24:32.155539 12104 ts_tablet_manager.cc:1397] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260927 03:24:32.157570 12104 raft_consensus.cc:348] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bbf5ea4ffd485784b874b69c5967ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34167 } } peers { permanent_uuid: "dfa29204d89744cc9528b20f75c8d21d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38517 } } peers { permanent_uuid: "7abf79ebadac489db14166e69b3581ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38137 } }
I20260927 03:24:32.157752 12104 raft_consensus.cc:374] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:24:32.157790 12104 raft_consensus.cc:729] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99bbf5ea4ffd485784b874b69c5967ac, State: Initialized, Role: FOLLOWER
I20260927 03:24:32.157924 12104 consensus_queue.cc:260] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bbf5ea4ffd485784b874b69c5967ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34167 } } peers { permanent_uuid: "dfa29204d89744cc9528b20f75c8d21d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38517 } } peers { permanent_uuid: "7abf79ebadac489db14166e69b3581ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38137 } }
I20260927 03:24:32.158232 12104 ts_tablet_manager.cc:1428] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:24:32.158356 12083 heartbeater.cc:503] Master 127.0.98.62:40863 was elected leader, sending a full tablet report...
Sep 27 03:24:32 dist-test-slave-gw2v krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:24:32.261107 12110 raft_consensus.cc:482] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:24:32.261245 12110 raft_consensus.cc:504] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bbf5ea4ffd485784b874b69c5967ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34167 } } peers { permanent_uuid: "dfa29204d89744cc9528b20f75c8d21d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38517 } } peers { permanent_uuid: "7abf79ebadac489db14166e69b3581ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38137 } }
I20260927 03:24:32.261648 12110 leader_election.cc:288] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa29204d89744cc9528b20f75c8d21d (127.0.98.1:38517), 7abf79ebadac489db14166e69b3581ab (127.0.98.2:38137)
I20260927 03:24:32.266093 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:32.261879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38517 (local address 127.0.98.3:47581)
0927 03:24:32.262218 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 03:24:32.262231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:24:32.262276 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:24:32.262673 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:24:32.262674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:24:32.262677 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:24:32.262745 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:32.262747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.263760 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.263762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:32.264973 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:32.264977 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.265839 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.265840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:32.265918 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:32.265922 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:24:32.265967 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:24:32.266013 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318}
I20260927 03:24:32.266533 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:32.262399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47581 (local address 127.0.98.1:38517)
0927 03:24:32.262444 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:24:32.262446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:24:32.262454 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:24:32.262523 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:24:32.262525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:24:32.262536 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:24:32.262584 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:24:32.262841 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.263645 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.265074 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.265712 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.265772 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:32.265785 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:24:32.266412 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 03:24:32.266457 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:24:32.267076 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3311ec07e0824495937a3161482d6e17" candidate_uuid: "99bbf5ea4ffd485784b874b69c5967ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa29204d89744cc9528b20f75c8d21d" is_pre_election: true
I20260927 03:24:32.267264 11765 raft_consensus.cc:2454] T 3311ec07e0824495937a3161482d6e17 P dfa29204d89744cc9528b20f75c8d21d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99bbf5ea4ffd485784b874b69c5967ac in term 0.
I20260927 03:24:32.267515 11971 leader_election.cc:302] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99bbf5ea4ffd485784b874b69c5967ac, dfa29204d89744cc9528b20f75c8d21d; no voters:
I20260927 03:24:32.267647 12110 raft_consensus.cc:2781] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:24:32.267688 12110 raft_consensus.cc:482] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:24:32.267706 12110 raft_consensus.cc:3037] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:24:32.268635 12110 raft_consensus.cc:504] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bbf5ea4ffd485784b874b69c5967ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34167 } } peers { permanent_uuid: "dfa29204d89744cc9528b20f75c8d21d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38517 } } peers { permanent_uuid: "7abf79ebadac489db14166e69b3581ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38137 } }
I20260927 03:24:32.268797 12110 leader_election.cc:288] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [CANDIDATE]: Term 1 election: Requested vote from peers dfa29204d89744cc9528b20f75c8d21d (127.0.98.1:38517), 7abf79ebadac489db14166e69b3581ab (127.0.98.2:38137)
I20260927 03:24:32.268996 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3311ec07e0824495937a3161482d6e17" candidate_uuid: "99bbf5ea4ffd485784b874b69c5967ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa29204d89744cc9528b20f75c8d21d"
I20260927 03:24:32.269074 11765 raft_consensus.cc:3037] T 3311ec07e0824495937a3161482d6e17 P dfa29204d89744cc9528b20f75c8d21d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:24:32.269899 11765 raft_consensus.cc:2454] T 3311ec07e0824495937a3161482d6e17 P dfa29204d89744cc9528b20f75c8d21d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99bbf5ea4ffd485784b874b69c5967ac in term 1.
I20260927 03:24:32.270099 11971 leader_election.cc:302] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99bbf5ea4ffd485784b874b69c5967ac, dfa29204d89744cc9528b20f75c8d21d; no voters:
I20260927 03:24:32.270511 12110 raft_consensus.cc:2781] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:24:32.270670 12110 raft_consensus.cc:686] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [term 1 LEADER]: Becoming Leader. State: Replica: 99bbf5ea4ffd485784b874b69c5967ac, State: Running, Role: LEADER
I20260927 03:24:32.270787 12110 consensus_queue.cc:237] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bbf5ea4ffd485784b874b69c5967ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34167 } } peers { permanent_uuid: "dfa29204d89744cc9528b20f75c8d21d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38517 } } peers { permanent_uuid: "7abf79ebadac489db14166e69b3581ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38137 } }
I20260927 03:24:32.272010 11599 catalog_manager.cc:5495] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 99bbf5ea4ffd485784b874b69c5967ac (127.0.98.3). New cstate: current_term: 1 leader_uuid: "99bbf5ea4ffd485784b874b69c5967ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99bbf5ea4ffd485784b874b69c5967ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa29204d89744cc9528b20f75c8d21d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7abf79ebadac489db14166e69b3581ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38137 } health_report { overall_health: UNKNOWN } } }
I20260927 03:24:32.277297 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:32.262463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39589 (local address 127.0.98.2:38137)
0927 03:24:32.262639 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 03:24:32.262641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:24:32.265900 (+ 3259us) server_negotiation.cc:407] Connection header received
0927 03:24:32.266018 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:24:32.266019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:24:32.266029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:24:32.266057 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:24:32.266395 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.270413 (+ 4018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.275915 (+ 5502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.276483 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.276536 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:32.276548 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:24:32.277175 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 03:24:32.277222 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260927 03:24:32.277567 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:32.261995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38137 (local address 127.0.98.3:39589)
0927 03:24:32.265836 (+ 3841us) negotiation.cc:107] Waiting for socket to connect
0927 03:24:32.265847 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:24:32.265928 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:24:32.266114 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:24:32.266115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:24:32.266116 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:24:32.266324 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:32.266326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.272395 (+ 6069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.272396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:32.275795 (+ 3399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:24:32.275800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.276997 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.276999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:24:32.277078 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:32.277082 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:24:32.277427 (+ 345us) client_negotiation.cc:240] Negotiation successful
0927 03:24:32.277475 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3818,"thread_start_us":99,"threads_started":1}
I20260927 03:24:32.277906 11896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3311ec07e0824495937a3161482d6e17" candidate_uuid: "99bbf5ea4ffd485784b874b69c5967ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abf79ebadac489db14166e69b3581ab" is_pre_election: true
I20260927 03:24:32.278057 11896 raft_consensus.cc:2454] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99bbf5ea4ffd485784b874b69c5967ac in term 0.
I20260927 03:24:32.278158 11895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3311ec07e0824495937a3161482d6e17" candidate_uuid: "99bbf5ea4ffd485784b874b69c5967ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abf79ebadac489db14166e69b3581ab"
I20260927 03:24:32.278203 11895 raft_consensus.cc:3037] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:24:32.279083 11895 raft_consensus.cc:2454] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99bbf5ea4ffd485784b874b69c5967ac in term 1.
W20260927 03:24:32.281452 11949 tablet.cc:2369] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:24:32.285310 11812 tablet.cc:2369] T 3311ec07e0824495937a3161482d6e17 P dfa29204d89744cc9528b20f75c8d21d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:24:32.320528 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:32.316988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.0.98.3:34167)
0927 03:24:32.317033 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:24:32.317035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:24:32.317043 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:24:32.317087 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:24:32.317088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:24:32.317127 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:24:32.317199 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:24:32.317419 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.318190 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.319353 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:24:32.319642 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:24:32.319708 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:24:32.319929 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:24:32.320071 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:24:32.320151 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:24:32.320410 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 03:24:32.320427 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 03:24:32.321942 12084 tablet.cc:2369] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:24:32.331089 11895 raft_consensus.cc:1261] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab [term 1 FOLLOWER]: Refusing update from remote peer 99bbf5ea4ffd485784b874b69c5967ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:24:32.331585 12112 consensus_queue.cc:1035] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abf79ebadac489db14166e69b3581ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:24:32.333618 11765 raft_consensus.cc:1261] T 3311ec07e0824495937a3161482d6e17 P dfa29204d89744cc9528b20f75c8d21d [term 1 FOLLOWER]: Refusing update from remote peer 99bbf5ea4ffd485784b874b69c5967ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:24:32.333923 12112 consensus_queue.cc:1035] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa29204d89744cc9528b20f75c8d21d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:24:32.341002 12116 mvcc.cc:205] Tried to move back new op lower bound from 7333803918664290304 to 7333803918421970944. Current Snapshot: MvccSnapshot[applied={T|T < 7333803918664290304}]
I20260927 03:24:32.342631 12115 mvcc.cc:205] Tried to move back new op lower bound from 7333803918664290304 to 7333803918421970944. Current Snapshot: MvccSnapshot[applied={T|T < 7333803918664290304}]
I20260927 03:24:32.343770 12118 mvcc.cc:205] Tried to move back new op lower bound from 7333803918664290304 to 7333803918421970944. Current Snapshot: MvccSnapshot[applied={T|T < 7333803918664290304}]
I20260927 03:24:32.345340 11599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:24:32.345441 11599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:24:32.346150 11599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b62b8179b1841509a2e748b453c0a9f: Sending DeleteTablet for 3 replicas of tablet 3311ec07e0824495937a3161482d6e17
I20260927 03:24:32.346493 12017 tablet_service.cc:1514] Processing DeleteTablet for tablet 3311ec07e0824495937a3161482d6e17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:24:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59226
I20260927 03:24:32.346882 11881 tablet_service.cc:1514] Processing DeleteTablet for tablet 3311ec07e0824495937a3161482d6e17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:24:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58258
I20260927 03:24:32.347473 11745 tablet_service.cc:1514] Processing DeleteTablet for tablet 3311ec07e0824495937a3161482d6e17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:24:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36168
I20260927 03:24:32.347693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 11680
I20260927 03:24:32.354928 12125 tablet_replica.cc:330] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab: stopping tablet replica
I20260927 03:24:32.355077 12125 raft_consensus.cc:2229] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:24:32.355199 12125 raft_consensus.cc:2258] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:24:32.355377 12124 tablet_replica.cc:330] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac: stopping tablet replica
I20260927 03:24:32.355490 12124 raft_consensus.cc:2229] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:24:32.355620 12124 raft_consensus.cc:2258] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:24:32.356137 12125 ts_tablet_manager.cc:1905] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:24:32.356777 12124 ts_tablet_manager.cc:1905] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:24:32.362938 12125 ts_tablet_manager.cc:1918] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:24:32.363045 12125 log.cc:1196] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/3311ec07e0824495937a3161482d6e17
I20260927 03:24:32.363298 12125 ts_tablet_manager.cc:1939] T 3311ec07e0824495937a3161482d6e17 P 7abf79ebadac489db14166e69b3581ab: Deleting consensus metadata
I20260927 03:24:32.363761 12124 ts_tablet_manager.cc:1918] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:24:32.363839 12124 log.cc:1196] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/3311ec07e0824495937a3161482d6e17
I20260927 03:24:32.363807 11584 catalog_manager.cc:4841] TS 7abf79ebadac489db14166e69b3581ab (127.0.98.2:38137): tablet 3311ec07e0824495937a3161482d6e17 (table test-table [id=928fb8c8c87b4f8b987bd195dccf46a3]) successfully deleted
I20260927 03:24:32.364089 12124 ts_tablet_manager.cc:1939] T 3311ec07e0824495937a3161482d6e17 P 99bbf5ea4ffd485784b874b69c5967ac: Deleting consensus metadata
I20260927 03:24:32.364511 11583 catalog_manager.cc:4841] TS 99bbf5ea4ffd485784b874b69c5967ac (127.0.98.3:34167): tablet 3311ec07e0824495937a3161482d6e17 (table test-table [id=928fb8c8c87b4f8b987bd195dccf46a3]) successfully deleted
W20260927 03:24:32.371115 11584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38517 (error 108)
I20260927 03:24:32.372189 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:24:32.372036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38517 (local address 127.0.0.1:36178)
0927 03:24:32.372091 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:24:32.372148 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 03:24:32.372180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 11817
W20260927 03:24:32.372316 11584 catalog_manager.cc:4584] TS dfa29204d89744cc9528b20f75c8d21d (127.0.98.1:38517): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3311ec07e0824495937a3161482d6e17: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38517: connect: Connection refused (error 111)
I20260927 03:24:32.378125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 11954
I20260927 03:24:32.384405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 11567
I20260927 03:24:32.391438 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 03:24:32.725863 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 03:24:32.725963 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 03:24:33.060401 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 03:24:33.060 UTC [11153] LOG: received smart shutdown request
2026-09-27 03:24:33.064 UTC [11153] LOG: background worker "logical replication launcher" (PID 11162) exited with exit code 1
2026-09-27 03:24:33.064 UTC [11157] LOG: shutting down
2026-09-27 03:24:33.219 UTC [11153] LOG: database system is shut down
2026-09-27T03:24:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45379 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:24:33 dist-test-slave-gw2v krb5kdc[12136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:24:33 dist-test-slave-gw2v krb5kdc[12136](info): set up 2 sockets
Sep 27 03:24:33 dist-test-slave-gw2v krb5kdc[12136](info): commencing operation
krb5kdc: starting...
W20260927 03:24:34.258483 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:24:34 dist-test-slave-gw2v krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:24:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:24:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 03:24:34.417151 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskzllPpO/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/postgres -l logfile start
2026-09-27 03:24:36.834 UTC [12166] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 03:24:36.835 UTC [12166] LOG: listening on IPv4 address "127.0.98.20", port 53491
2026-09-27 03:24:36.838 UTC [12166] LOG: listening on Unix socket "/tmp/.s.PGSQL.53491"
2026-09-27 03:24:36.865 UTC [12169] LOG: database system was shut down at 2026-09-27 03:24:35 UTC
2026-09-27 03:24:36.869 UTC [12166] LOG: database system is ready to accept connections
I20260927 03:24:37.734458 392 mini_postgres.cc:96] Postgres bound to 53491
2026-09-27 03:24:37.740 UTC [12177] FATAL: database "slave" does not exist
127.0.98.20:53491 - accepting connections
I20260927 03:24:37.740664 392 mini_ranger.cc:161] Starting Ranger...
I20260927 03:24:37.754908 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 03:24:38.257993 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 03:24:38.258078 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-admin
2026-09-27 03:24:38,532 [I] DB FLAVOR :POSTGRES
2026-09-27 03:24:38,533 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:24:38,533 [I] Checking connection..
2026-09-27 03:24:38,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 03:24:38,771 [I] Checking connection passed.
2026-09-27 03:24:38,771 [I] --------- Verifying version history table ---------
2026-09-27 03:24:38,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:24:39,021 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:24:39,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:24:39,253 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:24:39,253 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 03:24:39,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 03:24:39.492 UTC [12241] WARNING: there is no transaction in progress
2026-09-27 03:24:39,498 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 03:24:39,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:24:39,754 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 03:24:39,754 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:24:39,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 03:24:40,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 03:24:40,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gw2v.c.gcp-upstream.internal','N') ;"
2026-09-27 03:24:40,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:24:40,734 [I] Table x_portal_user does not exist in database ranger
2026-09-27 03:24:40,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:24:40,996 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 03:24:40,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:24:41,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:24:41,500 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 03:24:41,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 03:24:42.483 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:42.501 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:42.517 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:42.534 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:42.552 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:42.570 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:42.628 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:42.634 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:42.648 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:42.659 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:42.670 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:42.686 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:42.696 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:42.703 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:42.710 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:42.716 UTC [12349] WARNING: there is no transaction in progress
2026-09-27 03:24:43,395 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 03:24:43,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 03:24:43,634 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 03:24:43,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:24:43,902 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 03:24:43,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:24:44,151 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 03:24:44,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:24:44,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:24:44,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gw2v.c.gcp-upstream.internal';"
2026-09-27 03:24:44,879 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 03:24:44,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 03:24:45,121 [I] DB_PATCHES have already been applied
I20260927 03:24:45.126740 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskzllPpO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 03:24:45.126844 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 03:24:45.129699 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:52667
Sep 27, 2026 3:24:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:24:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:24:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:24:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:24:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:24:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:24:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 03:24:46.140195 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.400s user 0.000s sys 0.008s
Sep 27, 2026 3:25:05 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:25:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 3:25:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:25:05 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:25:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:25:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:25:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:25:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:25:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:25:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:25:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:25:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:25:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:25:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:25:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:25:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:25:05 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:25:06 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 03:25:06.636971 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 03:25:06 dist-test-slave-gw2v krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479506, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 03:25:06.721268 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 03:25:06.730808 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 03:25:07.590298 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 03:25:07.590378 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-kms
I20260927 03:25:07.590395 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:53491
2026-09-27 03:25:07,742 [I] DB FLAVOR :POSTGRES
2026-09-27 03:25:07,742 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:25:07,742 [I] Checking connection
2026-09-27 03:25:07,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 03:25:08,017 [I] connection success
2026-09-27 03:25:08,018 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 03:25:08,018 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 03:25:08,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 03:25:08,337 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 03:25:08,337 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:25:08,337 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 03:25:08,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53491/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 03:25:08,636 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 03:25:09.183184 392 mini_ranger_kms.cc:310] Created kms service
I20260927 03:25:09.371444 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 03:25:09.420713 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 03:25:09 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479506, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:25:09.827224 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 03:25:09.827342 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskzllPpO/build/release/bin/postgresql.jar:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskzllPpO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 03:25:09.827360 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 03:25:09.830787 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12529
I20260927 03:25:09.830905 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:59011
Sep 27, 2026 3:25:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:25:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:25:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:25:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:25:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:25:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 03:25:11.877345 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.156s user 0.001s sys 0.132s
I20260927 03:25:11.877426 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 03:25:11.877472 392 mini_ranger_kms.cc:401] 127.0.98.20:59011/kms/v1/keys
Sep 27 03:25:12 dist-test-slave-gw2v krb5kdc[12136](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790479512, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 03:25:12 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479512, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 03:25:13 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479512, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 03:25:15 dist-test-slave-gw2v krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:25:15.317408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40485
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40485
--ranger_config_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59011/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:15.397400 12580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:15.397620 12580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:15.397650 12580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:15.398783 12580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:25:15.398849 12580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:15.398870 12580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:15.398885 12580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:25:15.398901 12580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:25:15.400534 12580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:59011/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40485
--ranger_config_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:15.400767 12580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:15.400969 12580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:15.406700 12585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:15.406872 12586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:15.406723 12580 server_base.cc:1034] running on GCE node
W20260927 03:25:15.406700 12588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:15.407270 12580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:15.407610 12580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:15.408800 12580 hybrid_clock.cc:648] HybridClock initialized: now 1790479515408777 us; error 39 us; skew 500 ppm
Sep 27 03:25:15 dist-test-slave-gw2v krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:15.415122 12580 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:25:15.415670 12580 webserver.cc:460] Webserver started at http://127.0.98.62:34913/ using document root <none> and password file <none>
I20260927 03:25:15.415897 12580 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:15.415946 12580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:15.416069 12580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:25:15 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:25:15.462904 12580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "e49035e1796d48629bc078731aa0e76f"
format_stamp: "Formatted at 2026-09-27 03:25:15 on dist-test-slave-gw2v"
server_key: "0ba37f040839c355f02e6ed5172471dc"
server_key_iv: "d0d21b07375bc7e65df5c89d64bdd81f"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:15.463387 12580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "e49035e1796d48629bc078731aa0e76f"
format_stamp: "Formatted at 2026-09-27 03:25:15 on dist-test-slave-gw2v"
server_key: "0ba37f040839c355f02e6ed5172471dc"
server_key_iv: "d0d21b07375bc7e65df5c89d64bdd81f"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:15.465054 12580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:25:15.476452 12595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:15.476657 12580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:25:15.476758 12580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "e49035e1796d48629bc078731aa0e76f"
format_stamp: "Formatted at 2026-09-27 03:25:15 on dist-test-slave-gw2v"
server_key: "0ba37f040839c355f02e6ed5172471dc"
server_key_iv: "d0d21b07375bc7e65df5c89d64bdd81f"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:15.476850 12580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:15.487486 12580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:15.489287 12580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:15.489418 12580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:15.493666 12580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40485
I20260927 03:25:15.493739 12647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40485 every 8 connection(s)
I20260927 03:25:15.494076 12580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:25:15.494712 12648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:15.497014 12648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f: Bootstrap starting.
I20260927 03:25:15.497581 12648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:15.497841 12648 log.cc:824] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:15.498548 12648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f: No bootstrap required, opened a new log
I20260927 03:25:15.500751 12648 raft_consensus.cc:348] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49035e1796d48629bc078731aa0e76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40485 } }
I20260927 03:25:15.500902 12648 raft_consensus.cc:374] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:15.500924 12648 raft_consensus.cc:729] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49035e1796d48629bc078731aa0e76f, State: Initialized, Role: FOLLOWER
I20260927 03:25:15.501060 12648 consensus_queue.cc:260] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49035e1796d48629bc078731aa0e76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40485 } }
I20260927 03:25:15.501132 12648 raft_consensus.cc:388] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:25:15.501168 12648 raft_consensus.cc:482] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:25:15.501214 12648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:15.501770 12648 raft_consensus.cc:504] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49035e1796d48629bc078731aa0e76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40485 } }
I20260927 03:25:15.501857 12648 leader_election.cc:302] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e49035e1796d48629bc078731aa0e76f; no voters:
I20260927 03:25:15.502017 12648 leader_election.cc:288] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:25:15.502061 12651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:25:15.502224 12651 raft_consensus.cc:686] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 1 LEADER]: Becoming Leader. State: Replica: e49035e1796d48629bc078731aa0e76f, State: Running, Role: LEADER
I20260927 03:25:15.502291 12648 sys_catalog.cc:564] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:25:15.502367 12651 consensus_queue.cc:237] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49035e1796d48629bc078731aa0e76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40485 } }
I20260927 03:25:15.502727 12651 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49035e1796d48629bc078731aa0e76f. Latest consensus state: current_term: 1 leader_uuid: "e49035e1796d48629bc078731aa0e76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49035e1796d48629bc078731aa0e76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40485 } } }
I20260927 03:25:15.502992 12651 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:15.502924 12653 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e49035e1796d48629bc078731aa0e76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49035e1796d48629bc078731aa0e76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40485 } } }
I20260927 03:25:15.503250 12653 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:15.503304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 12580
I20260927 03:25:15.503419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:25:15.503414 12660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:25:15.503585 392 external_mini_cluster.cc:1428] Setting key 2189552e2213e97fda0444ff3d0e5bf6
I20260927 03:25:15.503876 12660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:25:15.503716 12648 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 03:25:15.510807 12660 catalog_manager.cc:1347] Generated new cluster ID: 95e7f4ce645b40d7b4b765de25555040
I20260927 03:25:15.511021 12660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:25:15 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:15.519397 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:15.504238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60856 (local address 127.0.98.62:40485)
0927 03:25:15.504425 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:15.504435 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:15.506511 (+ 2076us) server_negotiation.cc:407] Connection header received
0927 03:25:15.506915 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:15.506927 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:15.507048 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:15.507989 (+ 941us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:15.508364 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:15.509972 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:15.511106 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:15.511788 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:15.511905 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:15.514950 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:15.514955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:15.514956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:15.514968 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:15.517253 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:15.517649 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:15.517653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:15.517653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:15.517755 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:15.517948 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:15.517949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:15.517949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:15.518295 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:15.518440 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:15.518960 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 03:25:15.519110 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":57,"threads_started":1}
I20260927 03:25:15.543951 12660 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:25:15.544817 12660 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:25:15.553198 12660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f: Generated new TSK 0
I20260927 03:25:15.553507 12660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:25:16.206424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40485
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:16.308036 12693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:16.308308 12693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:16.308341 12693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:16.310005 12693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:16.310074 12693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:16.310114 12693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:25:16.312318 12693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:59011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:16.312577 12693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:16.312888 12693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:16.317801 12701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:16.318005 12693 server_base.cc:1034] running on GCE node
W20260927 03:25:16.317818 12699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:16.317823 12698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:16.318367 12693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:16.318604 12693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:16.319761 12693 hybrid_clock.cc:648] HybridClock initialized: now 1790479516319735 us; error 41 us; skew 500 ppm
Sep 27 03:25:16 dist-test-slave-gw2v krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:16.322978 12693 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:25:16.323598 12693 webserver.cc:460] Webserver started at http://127.0.98.1:38475/ using document root <none> and password file <none>
I20260927 03:25:16.323902 12693 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:16.323985 12693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:16.324131 12693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:25:16 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:25:16.342257 12693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "27d356b770144028a9b133f1e99c859a"
format_stamp: "Formatted at 2026-09-27 03:25:16 on dist-test-slave-gw2v"
server_key: "4f2c58df2c8e7896db3b694e688a5ed4"
server_key_iv: "a5dd5f29514e1eee2529cce89346c213"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:16.342808 12693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "27d356b770144028a9b133f1e99c859a"
format_stamp: "Formatted at 2026-09-27 03:25:16 on dist-test-slave-gw2v"
server_key: "4f2c58df2c8e7896db3b694e688a5ed4"
server_key_iv: "a5dd5f29514e1eee2529cce89346c213"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:16.347169 12693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
I20260927 03:25:16.362950 12708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:16.363197 12693 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260927 03:25:16.363333 12693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "27d356b770144028a9b133f1e99c859a"
format_stamp: "Formatted at 2026-09-27 03:25:16 on dist-test-slave-gw2v"
server_key: "4f2c58df2c8e7896db3b694e688a5ed4"
server_key_iv: "a5dd5f29514e1eee2529cce89346c213"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:16.363510 12693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:16.389484 12693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:16.391916 12693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:16.392136 12693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:16.392412 12693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:16.392759 12693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:16.392818 12693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:16.392848 12693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:16.392864 12693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:16.398765 12693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39313
I20260927 03:25:16.401618 12693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:25:16.401163 12823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39313 every 8 connection(s)
I20260927 03:25:16.405285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 12693
I20260927 03:25:16.405387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:25:16.405552 392 external_mini_cluster.cc:1428] Setting key 650672f506a452bcf111436442a074fe
Sep 27 03:25:16 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:16.421952 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:16.402683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44453 (local address 127.0.98.62:40485)
0927 03:25:16.411932 (+ 9249us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:16.411934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:16.411942 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:25:16.411968 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:16.411969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:16.411981 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:16.412057 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:16.412896 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:16.413735 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:16.415354 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:16.415670 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:16.415731 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:16.418464 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:16.418466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:16.418466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:16.418474 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:16.420282 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:16.420776 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:16.420778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:16.420778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:16.420852 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:16.421167 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:16.421168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:16.421169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:16.421321 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:16.421393 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:16.421754 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 03:25:16.421842 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9213,"thread_start_us":95,"threads_started":1}
I20260927 03:25:16.422443 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:16.402668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40485 (local address 127.0.98.1:44453)
0927 03:25:16.410946 (+ 8278us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:16.411020 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:16.411696 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:16.412195 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:16.412201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:16.412465 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:16.412754 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:16.412761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:16.413865 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:16.413866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:16.415259 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:16.415262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:16.415827 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:16.415828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:16.415970 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:16.416601 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:16.416611 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:16.418341 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:16.420377 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:16.420384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:16.420387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:16.420683 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:16.420942 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:16.420942 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:16.420943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:16.421090 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:16.421400 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:16.421407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:16.421592 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 03:25:16.421716 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 03:25:16.421896 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8213,"thread_start_us":85,"threads_started":1}
I20260927 03:25:16.423012 12824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40485
I20260927 03:25:16.423130 12824 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:16.423354 12824 heartbeater.cc:511] Master 127.0.98.62:40485 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 03:25:16.424062 12612 ts_manager.cc:194] Registered new tserver with Master: 27d356b770144028a9b133f1e99c859a (127.0.98.1:39313)
I20260927 03:25:16.424991 12612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:25:16.447587 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40485
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:25:16 dist-test-slave-gw2v krb5kdc[12136](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790479516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 03:25:16.564785 12830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:16.565080 12830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:16.565119 12830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:16.566977 12830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:16.567082 12830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:16.567143 12830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:25:16.569411 12830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:59011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:16.569828 12830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:16.570134 12830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:16.576609 12838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:16.576952 12837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:16.576627 12840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:16.577128 12830 server_base.cc:1034] running on GCE node
I20260927 03:25:16.577337 12830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:16.577584 12830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:16.578722 12830 hybrid_clock.cc:648] HybridClock initialized: now 1790479516578700 us; error 36 us; skew 500 ppm
Sep 27 03:25:16 dist-test-slave-gw2v krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:16.581992 12830 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:25:16.582568 12830 webserver.cc:460] Webserver started at http://127.0.98.2:44019/ using document root <none> and password file <none>
I20260927 03:25:16.582851 12830 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:16.582952 12830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:16.583104 12830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:25:16 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:25:16.602069 12830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "793545090d47415d874a4594fc75cee3"
format_stamp: "Formatted at 2026-09-27 03:25:16 on dist-test-slave-gw2v"
server_key: "63910fb8a750141c9a3f064003a11f82"
server_key_iv: "806955ff6fedafe871cdcf720499c4cf"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:16.602530 12830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "793545090d47415d874a4594fc75cee3"
format_stamp: "Formatted at 2026-09-27 03:25:16 on dist-test-slave-gw2v"
server_key: "63910fb8a750141c9a3f064003a11f82"
server_key_iv: "806955ff6fedafe871cdcf720499c4cf"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:16.604791 12830 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 03:25:16.625742 12848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:16.626993 12830 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260927 03:25:16.627110 12830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "793545090d47415d874a4594fc75cee3"
format_stamp: "Formatted at 2026-09-27 03:25:16 on dist-test-slave-gw2v"
server_key: "63910fb8a750141c9a3f064003a11f82"
server_key_iv: "806955ff6fedafe871cdcf720499c4cf"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:16.627210 12830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:16.642077 12830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:16.644521 12830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:16.644738 12830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:16.645051 12830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:16.645496 12830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:16.645547 12830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:16.645576 12830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:16.645592 12830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:16.652716 12830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40481
I20260927 03:25:16.653177 12830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:25:16.653682 12961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40481 every 8 connection(s)
I20260927 03:25:16.654299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 12830
I20260927 03:25:16.654393 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:25:16.654567 392 external_mini_cluster.cc:1428] Setting key 49bb25928d7a3e36b0152c6a298b35a8
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 03:25:16 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:16.674939 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:16.654184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46569 (local address 127.0.98.62:40485)
0927 03:25:16.654243 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:16.654245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:16.654431 (+ 186us) server_negotiation.cc:407] Connection header received
0927 03:25:16.655132 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:16.655134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:16.655145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:16.655174 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:16.656007 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:16.656795 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:16.664403 (+ 7608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:16.664624 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:16.664685 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:16.667657 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:16.667659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:16.667660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:16.667669 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:16.669138 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:16.669592 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:16.669594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:16.669594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:16.669651 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:16.669889 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:16.669890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:16.669890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:16.670010 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:16.670064 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:16.674677 (+ 4613us) server_negotiation.cc:299] Negotiation successful
0927 03:25:16.674805 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:25:16.675496 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:16.653921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40485 (local address 127.0.98.2:46569)
0927 03:25:16.654312 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:16.654355 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:16.655009 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:16.655370 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:16.655375 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:16.655618 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:16.655902 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:16.655909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:16.656907 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:16.656907 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:16.664281 (+ 7374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:16.664287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:16.664740 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:16.664741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:16.664881 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:16.665413 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:16.665422 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:16.667533 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:16.669231 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:16.669239 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:16.669242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:16.669513 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:16.669703 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:16.669704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:16.669704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:16.669826 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:16.670061 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:16.670068 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:16.674459 (+ 4391us) client_negotiation.cc:769] Sending connection context
0927 03:25:16.675052 (+ 593us) client_negotiation.cc:240] Negotiation successful
0927 03:25:16.675252 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":73,"threads_started":1}
I20260927 03:25:16.676149 12962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40485
I20260927 03:25:16.676263 12962 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:16.676469 12962 heartbeater.cc:511] Master 127.0.98.62:40485 requested a full tablet report, sending...
I20260927 03:25:16.676906 12612 ts_manager.cc:194] Registered new tserver with Master: 793545090d47415d874a4594fc75cee3 (127.0.98.2:40481)
I20260927 03:25:16.677419 12612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:25:16.684114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40485
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:16.776640 12967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:16.777020 12967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:16.777115 12967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:16.778532 12967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:16.778628 12967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:16.778715 12967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:25:16.781010 12967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:59011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:16.781456 12967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:16.781786 12967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:16.787134 12972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:16.787420 12973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:16.787758 12967 server_base.cc:1034] running on GCE node
W20260927 03:25:16.788005 12975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:16.788223 12967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:16.788489 12967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:16.789640 12967 hybrid_clock.cc:648] HybridClock initialized: now 1790479516789614 us; error 35 us; skew 500 ppm
Sep 27 03:25:16 dist-test-slave-gw2v krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:16.792105 12967 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:25:16.792553 12967 webserver.cc:460] Webserver started at http://127.0.98.3:40319/ using document root <none> and password file <none>
I20260927 03:25:16.792804 12967 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:16.792845 12967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:16.792956 12967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:25:16 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:25:16.805534 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "2702f8affda94a51b386ba04674060c5"
format_stamp: "Formatted at 2026-09-27 03:25:16 on dist-test-slave-gw2v"
server_key: "6f3442321ca947bd81dc2c3deec64279"
server_key_iv: "ab9544cd78c1ba07eb709e610992d69e"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:16.805984 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2702f8affda94a51b386ba04674060c5"
format_stamp: "Formatted at 2026-09-27 03:25:16 on dist-test-slave-gw2v"
server_key: "6f3442321ca947bd81dc2c3deec64279"
server_key_iv: "ab9544cd78c1ba07eb709e610992d69e"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:16.807718 12967 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260927 03:25:16.821909 12982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:16.822389 12967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:25:16.822556 12967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "2702f8affda94a51b386ba04674060c5"
format_stamp: "Formatted at 2026-09-27 03:25:16 on dist-test-slave-gw2v"
server_key: "6f3442321ca947bd81dc2c3deec64279"
server_key_iv: "ab9544cd78c1ba07eb709e610992d69e"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:16.822772 12967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:16.847208 12967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:16.849642 12967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:16.849819 12967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:16.850071 12967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:16.850512 12967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:16.850600 12967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:16.850677 12967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:16.850747 12967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:16.857241 12967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44591
I20260927 03:25:16.857590 12967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:25:16.858122 13095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44591 every 8 connection(s)
I20260927 03:25:16.863243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 12967
I20260927 03:25:16.863340 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:25:16.863503 392 external_mini_cluster.cc:1428] Setting key 451e681836836d97abf60617c4ec6853
Sep 27 03:25:16 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:16.867653 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:16.858583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33027 (local address 127.0.98.62:40485)
0927 03:25:16.858624 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:16.858626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:16.858636 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:25:16.859177 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:16.859178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:16.859197 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:16.859246 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:16.859843 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:16.860697 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:16.861707 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:16.861937 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:16.861997 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:16.864280 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:16.864281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:16.864282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:16.864290 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:16.865912 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:16.866453 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:16.866454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:16.866454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:16.866527 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:16.866889 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:16.866890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:16.866891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:16.867045 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:16.867096 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:16.867511 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 03:25:16.867581 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:25:16.868440 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:16.858314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40485 (local address 127.0.98.3:33027)
0927 03:25:16.858473 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:16.858515 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:16.859077 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:16.859317 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:16.859322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:16.859493 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:16.859725 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:16.859732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:16.860815 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:16.860816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:16.861625 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:16.861628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:16.862077 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:16.862078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:16.862179 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:16.862653 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:16.862663 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:16.864156 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:16.866007 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:16.866021 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:16.866024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:16.866352 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:16.866603 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:16.866603 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:16.866604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:16.866756 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:16.867144 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:16.867150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:16.867350 (+ 200us) client_negotiation.cc:769] Sending connection context
0927 03:25:16.868030 (+ 680us) client_negotiation.cc:240] Negotiation successful
0927 03:25:16.868205 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260927 03:25:16.869097 13096 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40485
I20260927 03:25:16.869247 13096 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:16.869441 13096 heartbeater.cc:511] Master 127.0.98.62:40485 requested a full tablet report, sending...
I20260927 03:25:16.869771 12612 ts_manager.cc:194] Registered new tserver with Master: 2702f8affda94a51b386ba04674060c5 (127.0.98.3:44591)
I20260927 03:25:16.870128 12612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33027
I20260927 03:25:16.875352 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:25:16.875972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 12693
I20260927 03:25:16.881515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 12830
I20260927 03:25:16.886814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 12967
I20260927 03:25:16.891718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 12580
I20260927 03:25:16.896209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40485
--webserver_interface=127.0.98.62
--webserver_port=34913
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40485
--ranger_config_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59011/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59011/kms with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:16.982729 13099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:16.982995 13099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:16.983017 13099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:16.984553 13099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:25:16.984614 13099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:16.984635 13099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:16.984654 13099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:25:16.984680 13099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:25:16.987180 13099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59011/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40485
--ranger_config_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:16.987452 13099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:16.987733 13099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:16.992908 13105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:16.992998 13104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:16.992939 13107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:16.996026 13099 server_base.cc:1034] running on GCE node
I20260927 03:25:16.996310 13099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:16.996645 13099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:16.997795 13099 hybrid_clock.cc:648] HybridClock initialized: now 1790479516997780 us; error 31 us; skew 500 ppm
Sep 27 03:25:16 dist-test-slave-gw2v krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:17.001255 13099 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:25:17.001827 13099 webserver.cc:460] Webserver started at http://127.0.98.62:34913/ using document root <none> and password file <none>
I20260927 03:25:17.002095 13099 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:17.002161 13099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:25:17 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:25:17.064699 13099 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:25:17.065598 13115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:17.065805 13099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:25:17.065907 13099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "e49035e1796d48629bc078731aa0e76f"
format_stamp: "Formatted at 2026-09-27 03:25:15 on dist-test-slave-gw2v"
server_key: "0ba37f040839c355f02e6ed5172471dc"
server_key_iv: "d0d21b07375bc7e65df5c89d64bdd81f"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:17.066280 13099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:17.099184 13099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:17.101087 13099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:17.101213 13099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:17.105365 13099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40485
I20260927 03:25:17.105396 13167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40485 every 8 connection(s)
I20260927 03:25:17.105791 13099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:25:17.107110 13168 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:25:17.107769 13168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f: Bootstrap starting.
I20260927 03:25:17.112392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 13099
I20260927 03:25:17.112723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39313
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40485
--webserver_port=38475
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59011/kms with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:25:17.115871 13168 log.cc:824] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:17.117538 13168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:25:17.117805 13168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f: Bootstrap complete.
I20260927 03:25:17.120054 13168 raft_consensus.cc:348] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49035e1796d48629bc078731aa0e76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40485 } }
I20260927 03:25:17.120281 13168 raft_consensus.cc:729] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49035e1796d48629bc078731aa0e76f, State: Initialized, Role: FOLLOWER
I20260927 03:25:17.120426 13168 consensus_queue.cc:260] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49035e1796d48629bc078731aa0e76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40485 } }
I20260927 03:25:17.120489 13168 raft_consensus.cc:388] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:25:17.120527 13168 raft_consensus.cc:482] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:25:17.120569 13168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:25:17.121546 13168 raft_consensus.cc:504] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49035e1796d48629bc078731aa0e76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40485 } }
I20260927 03:25:17.121632 13168 leader_election.cc:302] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e49035e1796d48629bc078731aa0e76f; no voters:
I20260927 03:25:17.121811 13168 leader_election.cc:288] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:25:17.121883 13172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:25:17.122025 13172 raft_consensus.cc:686] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [term 2 LEADER]: Becoming Leader. State: Replica: e49035e1796d48629bc078731aa0e76f, State: Running, Role: LEADER
I20260927 03:25:17.122081 13168 sys_catalog.cc:564] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:25:17.122118 13172 consensus_queue.cc:237] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49035e1796d48629bc078731aa0e76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40485 } }
I20260927 03:25:17.122359 13172 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49035e1796d48629bc078731aa0e76f. Latest consensus state: current_term: 2 leader_uuid: "e49035e1796d48629bc078731aa0e76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49035e1796d48629bc078731aa0e76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40485 } } }
I20260927 03:25:17.122377 13173 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e49035e1796d48629bc078731aa0e76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49035e1796d48629bc078731aa0e76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40485 } } }
I20260927 03:25:17.122421 13172 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:17.122443 13173 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:17.122591 13177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:25:17.123081 13177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:25:17.123202 13177 catalog_manager.cc:1259] Loaded cluster ID: 95e7f4ce645b40d7b4b765de25555040
I20260927 03:25:17.123234 13177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:25:17.123551 13168 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 03:25:17.124439 13177 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:25:17.124631 13177 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f: Loaded TSK: 0
I20260927 03:25:17.124852 13177 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:25:17.223517 13170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:17.223801 13170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:17.223839 13170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:17.225682 13170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:17.225734 13170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:17.225775 13170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:25:17.228091 13170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38475
--tserver_master_addrs=127.0.98.62:40485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:17.228366 13170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:17.228607 13170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:17.235312 13208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:17.235304 13205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:17.235682 13206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:17.236037 13170 server_base.cc:1034] running on GCE node
I20260927 03:25:17.236318 13170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:17.236611 13170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:17.237800 13170 hybrid_clock.cc:648] HybridClock initialized: now 1790479517237705 us; error 105 us; skew 500 ppm
Sep 27 03:25:17 dist-test-slave-gw2v krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:17.241361 13170 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:25:17.241926 13170 webserver.cc:460] Webserver started at http://127.0.98.1:38475/ using document root <none> and password file <none>
I20260927 03:25:17.242205 13170 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:17.242301 13170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:25:17 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:25:17.263185 13170 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260927 03:25:17.269198 13216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:17.269418 13170 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260927 03:25:17.269510 13170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "27d356b770144028a9b133f1e99c859a"
format_stamp: "Formatted at 2026-09-27 03:25:16 on dist-test-slave-gw2v"
server_key: "4f2c58df2c8e7896db3b694e688a5ed4"
server_key_iv: "a5dd5f29514e1eee2529cce89346c213"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:17.269884 13170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:17.279500 13170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:17.281517 13170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:17.281652 13170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:17.281873 13170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:17.282220 13170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:17.282255 13170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:17.282285 13170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:17.282306 13170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:17.288846 13170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39313
I20260927 03:25:17.289253 13170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:25:17.289826 13329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39313 every 8 connection(s)
I20260927 03:25:17.290556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 13170
I20260927 03:25:17.290930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40481
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40485
--webserver_port=44019
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59011/kms with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:25:17 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:17.303552 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:17.290051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40485 (local address 127.0.98.1:58981)
0927 03:25:17.290199 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:17.290234 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:17.290852 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:17.291383 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:17.291388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:17.291613 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:17.291882 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:17.291890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:17.293571 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:17.293572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:17.295200 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:17.295203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:17.296082 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:17.296084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:17.296233 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:17.296864 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:17.296874 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:17.298535 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:17.301557 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:17.301566 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:17.301569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:17.301912 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:17.302215 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:17.302216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:17.302216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:17.302373 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:17.302814 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:17.302846 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:17.303030 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 03:25:17.303148 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 03:25:17.303317 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":85,"threads_started":1}
I20260927 03:25:17.303706 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:17.290479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58981 (local address 127.0.98.62:40485)
0927 03:25:17.290766 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:17.290775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:17.290837 (+ 62us) server_negotiation.cc:407] Connection header received
0927 03:25:17.291090 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:17.291097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:17.291159 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:17.291318 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:17.292028 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:17.293442 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:17.295295 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:17.295959 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:17.296081 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:17.298672 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:17.298678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:17.298678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:17.298692 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:17.300810 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:17.302013 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:17.302017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:17.302018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:17.302138 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:17.302439 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:17.302440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:17.302440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:17.302741 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:17.302856 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:17.303216 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 03:25:17.303341 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":74,"threads_started":1}
I20260927 03:25:17.304991 13330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40485
I20260927 03:25:17.305097 13330 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:17.305295 13330 heartbeater.cc:511] Master 127.0.98.62:40485 requested a full tablet report, sending...
W20260927 03:25:17.306169 13330 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40485 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 03:25:17.306239 13330 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:17.306296 13330 heartbeater.cc:511] Master 127.0.98.62:40485 requested a full tablet report, sending...
I20260927 03:25:17.306865 13330 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:17.306945 13330 heartbeater.cc:511] Master 127.0.98.62:40485 requested a full tablet report, sending...
W20260927 03:25:17.307286 13330 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:25:17.390998 13334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:17.391310 13334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:17.391357 13334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:17.393143 13334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:17.393213 13334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:17.393265 13334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:25:17.395592 13334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44019
--tserver_master_addrs=127.0.98.62:40485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:17.395859 13334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:17.396198 13334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:17.401731 13340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:17.401768 13339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:17.401737 13342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:17.402201 13334 server_base.cc:1034] running on GCE node
I20260927 03:25:17.402423 13334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:17.402710 13334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:17.403831 13334 hybrid_clock.cc:648] HybridClock initialized: now 1790479517403842 us; error 69 us; skew 500 ppm
Sep 27 03:25:17 dist-test-slave-gw2v krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:17.406543 13334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:25:17.407109 13334 webserver.cc:460] Webserver started at http://127.0.98.2:44019/ using document root <none> and password file <none>
I20260927 03:25:17.407366 13334 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:17.407433 13334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:25:17 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:25:17.431582 13334 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260927 03:25:17.432577 13350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:17.432760 13334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:25:17.432869 13334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "793545090d47415d874a4594fc75cee3"
format_stamp: "Formatted at 2026-09-27 03:25:16 on dist-test-slave-gw2v"
server_key: "63910fb8a750141c9a3f064003a11f82"
server_key_iv: "806955ff6fedafe871cdcf720499c4cf"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:17.433372 13334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:17.454231 13334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:17.456251 13334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:17.456398 13334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:17.456633 13334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:17.456971 13334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:17.457007 13334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:17.457038 13334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:17.457062 13334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:17.463683 13334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40481
I20260927 03:25:17.463811 13463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40481 every 8 connection(s)
I20260927 03:25:17.464104 13334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:25:17.469157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 13334
I20260927 03:25:17.469485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44591
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40485
--webserver_port=40319
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59011/kms with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:25:17 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:17.476838 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:17.465415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43733 (local address 127.0.98.62:40485)
0927 03:25:17.465460 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:17.465462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:17.465740 (+ 278us) server_negotiation.cc:407] Connection header received
0927 03:25:17.466247 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:17.466248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:17.466265 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:17.466316 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:17.467124 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:17.467666 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:17.468708 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:17.468899 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:17.468963 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:17.473517 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:17.473519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:17.473519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:17.473528 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:17.475213 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:17.475690 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:17.475691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:17.475692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:17.475754 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:17.476054 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:17.476055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:17.476055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:17.476200 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:17.476281 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:17.476638 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 03:25:17.476724 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:25:17.477268 13466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:17.465446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40485 (local address 127.0.98.2:43733)
0927 03:25:17.465617 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:17.465656 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:17.466146 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:17.466391 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:17.466396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:17.466647 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:17.467000 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:17.467007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:17.467779 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:17.467780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:17.468635 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:17.468638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:17.471049 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:17.471051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:17.471211 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:17.471789 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:17.471801 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:17.473405 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:17.475307 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:17.475315 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:17.475318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:17.475605 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:17.475823 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:17.475823 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:17.475824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:17.475975 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:17.476289 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:17.476296 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:17.476488 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 03:25:17.476614 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 03:25:17.476795 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260927 03:25:17.478014 13464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40485
I20260927 03:25:17.478184 13464 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:17.478432 13464 heartbeater.cc:511] Master 127.0.98.62:40485 requested a full tablet report, sending...
W20260927 03:25:17.479212 13464 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40485 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 03:25:17.479305 13464 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:17.479378 13464 heartbeater.cc:511] Master 127.0.98.62:40485 requested a full tablet report, sending...
I20260927 03:25:17.479686 13464 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:17.479776 13464 heartbeater.cc:511] Master 127.0.98.62:40485 requested a full tablet report, sending...
W20260927 03:25:17.480088 13464 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:25:17.560901 13467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:17.561115 13467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:17.561139 13467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:17.562350 13467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:17.562398 13467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:17.562425 13467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:25:17.564414 13467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40319
--tserver_master_addrs=127.0.98.62:40485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:17.564692 13467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:17.564983 13467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:17.571326 13475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:17.571518 13467 server_base.cc:1034] running on GCE node
W20260927 03:25:17.571658 13473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:17.571817 13472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:17.572122 13467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:17.572418 13467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:17.575678 13467 hybrid_clock.cc:648] HybridClock initialized: now 1790479517575648 us; error 40 us; skew 500 ppm
Sep 27 03:25:17 dist-test-slave-gw2v krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:17.579027 13467 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:25:17.579708 13467 webserver.cc:460] Webserver started at http://127.0.98.3:40319/ using document root <none> and password file <none>
I20260927 03:25:17.580047 13467 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:17.580233 13467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:25:17 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:25:17.600737 13467 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:25:17.601523 13483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:17.601707 13467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:25:17.601832 13467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "2702f8affda94a51b386ba04674060c5"
format_stamp: "Formatted at 2026-09-27 03:25:16 on dist-test-slave-gw2v"
server_key: "6f3442321ca947bd81dc2c3deec64279"
server_key_iv: "ab9544cd78c1ba07eb709e610992d69e"
server_key_version: "kuduclusterkey@0"
I20260927 03:25:17.602319 13467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:17.637970 13467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:17.640409 13467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:17.640571 13467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:17.640821 13467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:17.641151 13467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:17.641188 13467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:17.641215 13467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:17.641232 13467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:17.648244 13467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44591
I20260927 03:25:17.648307 13597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44591 every 8 connection(s)
I20260927 03:25:17.648738 13467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:25:17.649188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 13467
I20260927 03:25:17.657790 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:17.649614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.0.98.62:40485)
0927 03:25:17.649662 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:17.649664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:17.650094 (+ 430us) server_negotiation.cc:407] Connection header received
0927 03:25:17.650172 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:17.650173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:17.650185 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:17.650246 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:17.650856 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:17.651605 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:17.652747 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:17.653030 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:17.653109 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:17.654926 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:17.654928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:17.654929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:17.654935 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:17.656574 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:17.656964 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:17.656965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:17.656965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:17.657024 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:17.657250 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:17.657251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:17.657251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:17.657387 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:17.657561 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:17.657600 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 03:25:17.657676 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 27 03:25:17 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:17.662487 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:17.649795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40485 (local address 127.0.98.3:51105)
0927 03:25:17.649942 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:17.649992 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:17.650929 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:17.651162 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:17.651167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:17.651392 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:17.651663 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:17.651669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:17.652539 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:17.652540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:17.654125 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:17.654128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:17.654648 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:17.654649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:17.654809 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:17.655844 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:17.655854 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:17.659039 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:17.660720 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:17.660728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:17.660731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:17.661054 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:17.661259 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:17.661260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:17.661260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:17.661408 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:17.661720 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:17.661726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:17.661943 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 03:25:17.662058 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 03:25:17.662229 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260927 03:25:17.662932 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:17.649757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51105 (local address 127.0.98.62:40485)
0927 03:25:17.649916 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:17.649917 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:17.650093 (+ 176us) server_negotiation.cc:407] Connection header received
0927 03:25:17.651010 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:17.651011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:17.651023 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:17.651082 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:17.651741 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:17.652433 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:17.654219 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:17.654500 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:17.654571 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:17.659150 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:17.659152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:17.659152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:17.659160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:17.660626 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:17.661136 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:17.661137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:17.661137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:17.661195 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:17.661475 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:17.661476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:17.661476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:17.661631 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:17.661694 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:17.662696 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0927 03:25:17.662800 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260927 03:25:17.663550 13598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40485
I20260927 03:25:17.663658 13598 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:17.663877 13598 heartbeater.cc:511] Master 127.0.98.62:40485 requested a full tablet report, sending...
W20260927 03:25:17.664513 13598 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40485 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 03:25:17.664590 13598 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:17.664649 13598 heartbeater.cc:511] Master 127.0.98.62:40485 requested a full tablet report, sending...
I20260927 03:25:17.664963 13598 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:17.665030 13598 heartbeater.cc:511] Master 127.0.98.62:40485 requested a full tablet report, sending...
W20260927 03:25:17.665305 13598 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 03:25:18 dist-test-slave-gw2v krb5kdc[12136](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790479518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:18.307477 13330 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:18.307653 13330 heartbeater.cc:511] Master 127.0.98.62:40485 requested a full tablet report, sending...
I20260927 03:25:18.308106 13132 ts_manager.cc:194] Registered new tserver with Master: 27d356b770144028a9b133f1e99c859a (127.0.98.1:39313)
I20260927 03:25:18.309074 13132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58981
I20260927 03:25:18.480283 13464 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:18.480449 13464 heartbeater.cc:511] Master 127.0.98.62:40485 requested a full tablet report, sending...
I20260927 03:25:18.480883 13132 ts_manager.cc:194] Registered new tserver with Master: 793545090d47415d874a4594fc75cee3 (127.0.98.2:40481)
I20260927 03:25:18.481352 13132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43733
Sep 27 03:25:18 dist-test-slave-gw2v krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:25:18.665473 13598 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:18.665637 13598 heartbeater.cc:511] Master 127.0.98.62:40485 requested a full tablet report, sending...
I20260927 03:25:18.665939 13132 ts_manager.cc:194] Registered new tserver with Master: 2702f8affda94a51b386ba04674060c5 (127.0.98.3:44591)
I20260927 03:25:18.666432 13132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51105
I20260927 03:25:18.672487 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:25:18.686522 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:18.673641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60862 (local address 127.0.98.62:40485)
0927 03:25:18.673813 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:18.673815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:18.679084 (+ 5269us) server_negotiation.cc:407] Connection header received
0927 03:25:18.679159 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:18.679160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:18.679176 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:18.679252 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:18.679953 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.680779 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.681755 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.682081 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.682164 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:18.683590 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:18.683591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:18.683592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:18.683601 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:18.685267 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:18.685640 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:18.685641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:18.685642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:18.685710 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:18.685877 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:18.685877 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:18.685877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:18.686011 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:18.686105 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:18.686218 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 03:25:18.686299 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260927 03:25:18.688598 13132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:25:18.690924 13132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:25:18.698032 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:18.694039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.0.98.2:40481)
0927 03:25:18.694169 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:18.694175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:18.694188 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:25:18.694280 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:18.694282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:18.694301 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:18.694386 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:18.694729 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.695654 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.696858 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.697577 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.697619 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:18.697689 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:18.697906 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 03:25:18.697948 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":49,"threads_started":1}
I20260927 03:25:18.698032 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:18.693915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40481 (local address 127.0.0.1:59296)
0927 03:25:18.694080 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:18.694121 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:18.694185 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:18.694529 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:18.694533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:18.694536 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:18.694614 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:18.694617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.695742 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.695743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:18.696747 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:18.696749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.697741 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.697742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:18.697800 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:18.697810 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 03:25:18.697872 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 03:25:18.697932 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260927 03:25:18.698634 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:18.693934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44591 (local address 127.0.0.1:49630)
0927 03:25:18.694496 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:18.694507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:18.694550 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:18.694993 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:18.694994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:18.694997 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:18.695040 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:18.695041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.696300 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.696301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:18.697343 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:18.697346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.698398 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.698400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:18.698465 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:18.698470 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:25:18.698527 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:25:18.698566 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":91,"threads_started":1}
I20260927 03:25:18.698922 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:18.694403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49630 (local address 127.0.98.3:44591)
0927 03:25:18.694726 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:18.694729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:18.694739 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:25:18.694773 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:18.694775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:18.694792 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:18.694906 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:18.695121 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.696204 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.697451 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.698213 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.698285 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:18.698363 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:18.698562 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 03:25:18.698612 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":47,"threads_started":1}
I20260927 03:25:18.699123 13398 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad52c71055f4163a46149382e4861c0 (DEFAULT_TABLE table=test-table [id=b2941779d0594a5f98c41b9c7c7ae965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:25:18.699512 13398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad52c71055f4163a46149382e4861c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:18.699710 13532 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad52c71055f4163a46149382e4861c0 (DEFAULT_TABLE table=test-table [id=b2941779d0594a5f98c41b9c7c7ae965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:25:18.700016 13532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad52c71055f4163a46149382e4861c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:18.701463 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:18.694488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.0.98.1:39313)
0927 03:25:18.694656 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:18.694660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:18.698351 (+ 3691us) server_negotiation.cc:407] Connection header received
0927 03:25:18.698396 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:18.698399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:18.698415 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:18.698488 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:18.698701 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.699636 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.700608 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.701141 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.701218 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:18.701291 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:18.701350 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 03:25:18.701389 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260927 03:25:18.701452 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:18.694194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39313 (local address 127.0.0.1:33966)
0927 03:25:18.698236 (+ 4042us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:18.698246 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:18.698284 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:18.698580 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:18.698581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:18.698583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:18.698617 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:18.698618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.699714 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.699715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:18.700520 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:18.700522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.701224 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.701225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:18.701284 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:18.701288 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:25:18.701338 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:25:18.701379 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4030,"thread_start_us":67,"threads_started":1}
I20260927 03:25:18.703022 13631 tablet_bootstrap.cc:492] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3: Bootstrap starting.
I20260927 03:25:18.703560 13264 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad52c71055f4163a46149382e4861c0 (DEFAULT_TABLE table=test-table [id=b2941779d0594a5f98c41b9c7c7ae965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:25:18.703727 13631 tablet_bootstrap.cc:654] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:18.703881 13264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad52c71055f4163a46149382e4861c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:18.703997 13631 log.cc:824] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:18.704921 13631 tablet_bootstrap.cc:492] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3: No bootstrap required, opened a new log
I20260927 03:25:18.704995 13631 ts_tablet_manager.cc:1397] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:25:18.708072 13631 raft_consensus.cc:348] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793545090d47415d874a4594fc75cee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40481 } } peers { permanent_uuid: "2702f8affda94a51b386ba04674060c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44591 } } peers { permanent_uuid: "27d356b770144028a9b133f1e99c859a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } }
I20260927 03:25:18.708245 13631 raft_consensus.cc:374] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:18.708288 13631 raft_consensus.cc:729] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793545090d47415d874a4594fc75cee3, State: Initialized, Role: FOLLOWER
I20260927 03:25:18.708418 13631 consensus_queue.cc:260] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793545090d47415d874a4594fc75cee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40481 } } peers { permanent_uuid: "2702f8affda94a51b386ba04674060c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44591 } } peers { permanent_uuid: "27d356b770144028a9b133f1e99c859a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } }
I20260927 03:25:18.708663 13631 ts_tablet_manager.cc:1428] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 03:25:18.708786 13464 heartbeater.cc:503] Master 127.0.98.62:40485 was elected leader, sending a full tablet report...
I20260927 03:25:18.708917 13634 tablet_bootstrap.cc:492] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a: Bootstrap starting.
I20260927 03:25:18.709645 13634 tablet_bootstrap.cc:654] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:18.709951 13634 log.cc:824] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:18.710676 13634 tablet_bootstrap.cc:492] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a: No bootstrap required, opened a new log
I20260927 03:25:18.710736 13634 ts_tablet_manager.cc:1397] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:25:18.711014 13632 tablet_bootstrap.cc:492] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5: Bootstrap starting.
I20260927 03:25:18.711705 13632 tablet_bootstrap.cc:654] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:18.712051 13632 log.cc:824] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:18.712777 13632 tablet_bootstrap.cc:492] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5: No bootstrap required, opened a new log
I20260927 03:25:18.712839 13632 ts_tablet_manager.cc:1397] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:25:18.713358 13634 raft_consensus.cc:348] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793545090d47415d874a4594fc75cee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40481 } } peers { permanent_uuid: "2702f8affda94a51b386ba04674060c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44591 } } peers { permanent_uuid: "27d356b770144028a9b133f1e99c859a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } }
I20260927 03:25:18.713544 13634 raft_consensus.cc:374] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:18.713577 13634 raft_consensus.cc:729] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d356b770144028a9b133f1e99c859a, State: Initialized, Role: FOLLOWER
I20260927 03:25:18.713748 13634 consensus_queue.cc:260] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793545090d47415d874a4594fc75cee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40481 } } peers { permanent_uuid: "2702f8affda94a51b386ba04674060c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44591 } } peers { permanent_uuid: "27d356b770144028a9b133f1e99c859a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } }
I20260927 03:25:18.714309 13634 ts_tablet_manager.cc:1428] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260927 03:25:18.714419 13330 heartbeater.cc:503] Master 127.0.98.62:40485 was elected leader, sending a full tablet report...
I20260927 03:25:18.715288 13632 raft_consensus.cc:348] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793545090d47415d874a4594fc75cee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40481 } } peers { permanent_uuid: "2702f8affda94a51b386ba04674060c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44591 } } peers { permanent_uuid: "27d356b770144028a9b133f1e99c859a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } }
I20260927 03:25:18.715447 13632 raft_consensus.cc:374] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:18.715479 13632 raft_consensus.cc:729] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2702f8affda94a51b386ba04674060c5, State: Initialized, Role: FOLLOWER
I20260927 03:25:18.715576 13632 consensus_queue.cc:260] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793545090d47415d874a4594fc75cee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40481 } } peers { permanent_uuid: "2702f8affda94a51b386ba04674060c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44591 } } peers { permanent_uuid: "27d356b770144028a9b133f1e99c859a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } }
I20260927 03:25:18.715795 13632 ts_tablet_manager.cc:1428] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:25:18.715821 13598 heartbeater.cc:503] Master 127.0.98.62:40485 was elected leader, sending a full tablet report...
W20260927 03:25:18.716018 13465 tablet.cc:2369] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:25:18.725541 13635 raft_consensus.cc:482] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:25:18.725708 13635 raft_consensus.cc:504] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793545090d47415d874a4594fc75cee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40481 } } peers { permanent_uuid: "2702f8affda94a51b386ba04674060c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44591 } } peers { permanent_uuid: "27d356b770144028a9b133f1e99c859a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } }
I20260927 03:25:18.726156 13635 leader_election.cc:288] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2702f8affda94a51b386ba04674060c5 (127.0.98.3:44591), 27d356b770144028a9b133f1e99c859a (127.0.98.1:39313)
I20260927 03:25:18.730652 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:18.726631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38859 (local address 127.0.98.1:39313)
0927 03:25:18.726680 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:18.726682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:18.726884 (+ 202us) server_negotiation.cc:407] Connection header received
0927 03:25:18.727016 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:18.727017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:18.727029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:18.727061 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:18.727392 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.728141 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.729632 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.730225 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.730276 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:18.730290 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:18.730521 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 03:25:18.730570 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:25:18.730976 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:18.726406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39313 (local address 127.0.98.2:38859)
0927 03:25:18.726776 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:18.726787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:18.726920 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:18.727135 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:18.727136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:18.727140 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:18.727280 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:18.727282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.728244 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.728245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:18.729515 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:18.729518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.730357 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.730359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:18.730427 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:18.730432 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:25:18.730807 (+ 375us) client_negotiation.cc:240] Negotiation successful
0927 03:25:18.730884 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
I20260927 03:25:18.731168 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:18.726629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57731 (local address 127.0.98.3:44591)
0927 03:25:18.726691 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:18.726693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:18.726910 (+ 217us) server_negotiation.cc:407] Connection header received
0927 03:25:18.727015 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:18.727016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:18.727028 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:18.727061 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:18.727894 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.728535 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.729779 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.730336 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.730377 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:18.730386 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:18.730631 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 03:25:18.730676 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:25:18.731312 13284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad52c71055f4163a46149382e4861c0" candidate_uuid: "793545090d47415d874a4594fc75cee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d356b770144028a9b133f1e99c859a" is_pre_election: true
I20260927 03:25:18.731504 13284 raft_consensus.cc:2454] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793545090d47415d874a4594fc75cee3 in term 0.
I20260927 03:25:18.731675 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:18.726419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44591 (local address 127.0.98.2:57731)
0927 03:25:18.726853 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:18.726862 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:18.726932 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:18.727135 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:18.727136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:18.727137 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:18.727787 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:18.727792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.728635 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.728636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:18.729683 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:18.729686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.730483 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.730485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:18.730549 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:18.730552 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:25:18.731363 (+ 811us) client_negotiation.cc:240] Negotiation successful
0927 03:25:18.731418 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":29,"thread_start_us":105,"threads_started":1}
I20260927 03:25:18.731840 13354 leader_election.cc:302] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27d356b770144028a9b133f1e99c859a, 793545090d47415d874a4594fc75cee3; no voters:
I20260927 03:25:18.732172 13635 raft_consensus.cc:2781] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:25:18.732240 13635 raft_consensus.cc:482] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:25:18.732272 13635 raft_consensus.cc:3037] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:18.733038 13635 raft_consensus.cc:504] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793545090d47415d874a4594fc75cee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40481 } } peers { permanent_uuid: "2702f8affda94a51b386ba04674060c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44591 } } peers { permanent_uuid: "27d356b770144028a9b133f1e99c859a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } }
I20260927 03:25:18.733180 13635 leader_election.cc:288] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [CANDIDATE]: Term 1 election: Requested vote from peers 2702f8affda94a51b386ba04674060c5 (127.0.98.3:44591), 27d356b770144028a9b133f1e99c859a (127.0.98.1:39313)
I20260927 03:25:18.733343 13552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad52c71055f4163a46149382e4861c0" candidate_uuid: "793545090d47415d874a4594fc75cee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2702f8affda94a51b386ba04674060c5" is_pre_election: true
I20260927 03:25:18.733493 13552 raft_consensus.cc:2454] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793545090d47415d874a4594fc75cee3 in term 0.
I20260927 03:25:18.733861 13552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad52c71055f4163a46149382e4861c0" candidate_uuid: "793545090d47415d874a4594fc75cee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2702f8affda94a51b386ba04674060c5"
I20260927 03:25:18.733944 13552 raft_consensus.cc:3037] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:18.734699 13552 raft_consensus.cc:2454] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793545090d47415d874a4594fc75cee3 in term 1.
I20260927 03:25:18.734939 13352 leader_election.cc:302] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2702f8affda94a51b386ba04674060c5, 793545090d47415d874a4594fc75cee3; no voters:
I20260927 03:25:18.734938 13284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad52c71055f4163a46149382e4861c0" candidate_uuid: "793545090d47415d874a4594fc75cee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27d356b770144028a9b133f1e99c859a"
I20260927 03:25:18.735034 13284 raft_consensus.cc:3037] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:18.735749 13284 raft_consensus.cc:2454] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793545090d47415d874a4594fc75cee3 in term 1.
I20260927 03:25:18.735935 13635 raft_consensus.cc:2781] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:25:18.736125 13635 raft_consensus.cc:686] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [term 1 LEADER]: Becoming Leader. State: Replica: 793545090d47415d874a4594fc75cee3, State: Running, Role: LEADER
I20260927 03:25:18.736231 13635 consensus_queue.cc:237] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793545090d47415d874a4594fc75cee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40481 } } peers { permanent_uuid: "2702f8affda94a51b386ba04674060c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44591 } } peers { permanent_uuid: "27d356b770144028a9b133f1e99c859a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } }
I20260927 03:25:18.737103 13132 catalog_manager.cc:5495] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793545090d47415d874a4594fc75cee3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "793545090d47415d874a4594fc75cee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793545090d47415d874a4594fc75cee3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2702f8affda94a51b386ba04674060c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27d356b770144028a9b133f1e99c859a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } health_report { overall_health: UNKNOWN } } }
I20260927 03:25:18.747886 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:18.745632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.0.98.2:40481)
0927 03:25:18.745681 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:18.745682 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:18.745689 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:25:18.745723 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:18.745724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:18.745746 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:18.745805 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:25:18.745933 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.746399 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.747196 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:18.747399 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:18.747451 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:18.747590 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:25:18.747694 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:25:18.747765 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:18.747787 (+ 22us) server_negotiation.cc:299] Negotiation successful
0927 03:25:18.747809 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:25:18.749480 13284 raft_consensus.cc:1261] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a [term 1 FOLLOWER]: Refusing update from remote peer 793545090d47415d874a4594fc75cee3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:25:18.749809 13635 consensus_queue.cc:1035] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d356b770144028a9b133f1e99c859a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:25:18.749953 13552 raft_consensus.cc:1261] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5 [term 1 FOLLOWER]: Refusing update from remote peer 793545090d47415d874a4594fc75cee3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:25:18.750239 13635 consensus_queue.cc:1035] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2702f8affda94a51b386ba04674060c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:25:18.753064 13646 mvcc.cc:205] Tried to move back new op lower bound from 7333804108795744256 to 7333804108744351744. Current Snapshot: MvccSnapshot[applied={T|T < 7333804108795744256}]
I20260927 03:25:18.753257 13645 mvcc.cc:205] Tried to move back new op lower bound from 7333804108795744256 to 7333804108744351744. Current Snapshot: MvccSnapshot[applied={T|T < 7333804108795744256}]
I20260927 03:25:18.755622 13132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:25:18.755738 13132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:25:18.755873 13648 mvcc.cc:205] Tried to move back new op lower bound from 7333804108795744256 to 7333804108744351744. Current Snapshot: MvccSnapshot[applied={T|T < 7333804108795744256}]
I20260927 03:25:18.756433 13132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e49035e1796d48629bc078731aa0e76f: Sending DeleteTablet for 3 replicas of tablet 1ad52c71055f4163a46149382e4861c0
I20260927 03:25:18.756743 13532 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad52c71055f4163a46149382e4861c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:25:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49630
I20260927 03:25:18.757171 13264 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad52c71055f4163a46149382e4861c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:25:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33966
I20260927 03:25:18.757129 13398 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad52c71055f4163a46149382e4861c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:25:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59296
I20260927 03:25:18.757700 13655 tablet_replica.cc:330] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a: stopping tablet replica
I20260927 03:25:18.757745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 13170
I20260927 03:25:18.757793 13655 raft_consensus.cc:2229] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:25:18.757892 13655 raft_consensus.cc:2258] T 1ad52c71055f4163a46149382e4861c0 P 27d356b770144028a9b133f1e99c859a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:25:18.758538 13654 tablet_replica.cc:330] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5: stopping tablet replica
I20260927 03:25:18.758642 13654 raft_consensus.cc:2229] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:25:18.758708 13654 raft_consensus.cc:2258] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:25:18.759620 13654 ts_tablet_manager.cc:1905] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:25:18.761300 13654 ts_tablet_manager.cc:1918] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:25:18.761359 13654 log.cc:1196] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/1ad52c71055f4163a46149382e4861c0
I20260927 03:25:18.761603 13654 ts_tablet_manager.cc:1939] T 1ad52c71055f4163a46149382e4861c0 P 2702f8affda94a51b386ba04674060c5: Deleting consensus metadata
I20260927 03:25:18.761988 13117 catalog_manager.cc:4841] TS 2702f8affda94a51b386ba04674060c5 (127.0.98.3:44591): tablet 1ad52c71055f4163a46149382e4861c0 (table test-table [id=b2941779d0594a5f98c41b9c7c7ae965]) successfully deleted
I20260927 03:25:18.762956 13656 tablet_replica.cc:330] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3: stopping tablet replica
I20260927 03:25:18.763062 13656 raft_consensus.cc:2229] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:25:18.763150 13656 raft_consensus.cc:2258] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:25:18.763569 13656 ts_tablet_manager.cc:1905] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:25:18.765107 13656 ts_tablet_manager.cc:1918] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:25:18.765168 13656 log.cc:1196] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/1ad52c71055f4163a46149382e4861c0
W20260927 03:25:18.765156 13119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39313 (error 108)
I20260927 03:25:18.765424 13656 ts_tablet_manager.cc:1939] T 1ad52c71055f4163a46149382e4861c0 P 793545090d47415d874a4594fc75cee3: Deleting consensus metadata
I20260927 03:25:18.765878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 13334
I20260927 03:25:18.765862 13119 catalog_manager.cc:4841] TS 793545090d47415d874a4594fc75cee3 (127.0.98.2:40481): tablet 1ad52c71055f4163a46149382e4861c0 (table test-table [id=b2941779d0594a5f98c41b9c7c7ae965]) successfully deleted
I20260927 03:25:18.770577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 13467
I20260927 03:25:18.770738 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:18.770584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39313 (local address 127.0.0.1:33968)
0927 03:25:18.770648 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:18.770697 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 03:25:18.771416 13119 catalog_manager.cc:4584] TS 27d356b770144028a9b133f1e99c859a (127.0.98.1:39313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ad52c71055f4163a46149382e4861c0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39313: connect: Connection refused (error 111)
I20260927 03:25:18.775660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 13099
I20260927 03:25:18.780884 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 03:25:19.125736 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 03:25:19.125834 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 03:25:19.461129 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 03:25:19.461 UTC [12166] LOG: received smart shutdown request
2026-09-27 03:25:19.465 UTC [12166] LOG: background worker "logical replication launcher" (PID 12175) exited with exit code 1
2026-09-27 03:25:19.465 UTC [12170] LOG: shutting down
2026-09-27 03:25:19.644 UTC [12166] LOG: database system is shut down
2026-09-27T03:25:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46434 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:25:19 dist-test-slave-gw2v krb5kdc[13666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:25:19 dist-test-slave-gw2v krb5kdc[13666](info): set up 2 sockets
Sep 27 03:25:19 dist-test-slave-gw2v krb5kdc[13666](info): commencing operation
krb5kdc: starting...
W20260927 03:25:20.694481 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:25:20 dist-test-slave-gw2v krb5kdc[13666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:25:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:25:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:25:20.810040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43377
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43377,127.0.98.61:38385,127.0.98.60:34615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:20.889683 13679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:20.889899 13679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:20.889928 13679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:20.891125 13679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:25:20.891171 13679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:20.891191 13679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:20.891206 13679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:25:20.891222 13679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:25:20.892745 13679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43377,127.0.98.61:38385,127.0.98.60:34615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:20.892992 13679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:20.893200 13679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:25:20.898041 13679 server_base.cc:1034] running on GCE node
W20260927 03:25:20.897929 13684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:20.897961 13687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:20.897945 13685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:20.898541 13679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:20.898793 13679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:20.899959 13679 hybrid_clock.cc:648] HybridClock initialized: now 1790479520899948 us; error 29 us; skew 500 ppm
Sep 27 03:25:20 dist-test-slave-gw2v krb5kdc[13666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:20.906960 13679 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:25:20.907413 13679 webserver.cc:460] Webserver started at http://127.0.98.62:33007/ using document root <none> and password file <none>
I20260927 03:25:20.907595 13679 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:20.907635 13679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:20.907722 13679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:25:20.908751 13679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "ba5e602383804aeea278ca8aceb106d7"
format_stamp: "Formatted at 2026-09-27 03:25:20 on dist-test-slave-gw2v"
server_key: "1efee4d79332c2cfa60d8089aa42284b"
server_key_iv: "d7d6d9888ca36bbebb8dd6b14c419497"
server_key_version: "encryptionkey@0"
I20260927 03:25:20.909058 13679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba5e602383804aeea278ca8aceb106d7"
format_stamp: "Formatted at 2026-09-27 03:25:20 on dist-test-slave-gw2v"
server_key: "1efee4d79332c2cfa60d8089aa42284b"
server_key_iv: "d7d6d9888ca36bbebb8dd6b14c419497"
server_key_version: "encryptionkey@0"
I20260927 03:25:20.910305 13679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:25:20.911103 13694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:20.911266 13679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:25:20.911341 13679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "ba5e602383804aeea278ca8aceb106d7"
format_stamp: "Formatted at 2026-09-27 03:25:20 on dist-test-slave-gw2v"
server_key: "1efee4d79332c2cfa60d8089aa42284b"
server_key_iv: "d7d6d9888ca36bbebb8dd6b14c419497"
server_key_version: "encryptionkey@0"
I20260927 03:25:20.911422 13679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:20.940439 13679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:20.942641 13679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:20.942759 13679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:20.946735 13679 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43377
I20260927 03:25:20.946771 13746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43377 every 8 connection(s)
I20260927 03:25:20.947127 13679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:25:20.948000 13747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:20.949707 13747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:25:20.955307 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 13679
I20260927 03:25:20.955415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:25:20.955608 392 external_mini_cluster.cc:1428] Setting key 34d4cefdb918e8e58c27aaa380680261
Sep 27 03:25:20 dist-test-slave-gw2v krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:20.959976 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:20.950511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43377 (local address 127.0.0.1:34752)
0927 03:25:20.950695 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:20.950734 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:20.951228 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:20.951600 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:20.951603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:20.951864 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:20.952042 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:20.952047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:20.953044 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:20.953045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:20.953754 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:20.953756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:20.954270 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:20.954271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:20.954402 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:20.954937 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:20.954945 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:20.956642 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:20.958422 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:20.958429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:20.958431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:20.958670 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:20.958964 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:20.958966 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:20.958966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:20.959048 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:20.959434 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:20.959439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:20.959496 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 03:25:20.959586 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 03:25:20.959801 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":71,"thread_start_us":89,"threads_started":1}
I20260927 03:25:20.959997 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:20.950697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34752 (local address 127.0.98.62:43377)
0927 03:25:20.950861 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:20.950866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:20.950902 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:25:20.951479 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:20.951482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:20.951501 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:20.951587 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:20.952125 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:20.952960 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:20.953844 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:20.954181 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:20.954278 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:20.956736 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:20.956744 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:20.956744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:20.956752 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:20.958328 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:20.958749 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:20.958752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:20.958753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:20.958899 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:20.959103 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:20.959104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:20.959104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:20.959316 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:20.959392 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:20.959621 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 03:25:20.959731 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":50,"threads_started":1}
I20260927 03:25:20.960747 13747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:25:20.961122 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:20.961021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38385 (local address 127.0.0.1:54662)
0927 03:25:20.961061 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:20.961102 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 03:25:20.961238 13695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38385: connect: Connection refused (error 111)
I20260927 03:25:20.961598 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:20.961517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38385 (local address 127.0.0.1:54668)
0927 03:25:20.961553 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:20.961576 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 03:25:20.961689 13747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38385: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38385: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 03:25:20.980005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38385
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43377,127.0.98.61:38385,127.0.98.60:34615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:25:21.010960 13747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } attempt: 1
I20260927 03:25:21.011453 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.011313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38385 (local address 127.0.0.1:54676)
0927 03:25:21.011375 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.011416 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 03:25:21.011880 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.011732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38385 (local address 127.0.0.1:54680)
0927 03:25:21.011816 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.011852 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 03:25:21.011991 13747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38385: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38385: connect: Connection refused (error 111)
W20260927 03:25:21.060698 13754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:21.060920 13754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:21.060941 13754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:21.062114 13754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:25:21.062155 13754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:21.062167 13754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:21.062178 13754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:25:21.062188 13754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:25:21.063735 13754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43377,127.0.98.61:38385,127.0.98.60:34615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:21.064013 13754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:21.064248 13754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:21.069226 13762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:21.069190 13759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:21.069218 13760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:21.069475 13754 server_base.cc:1034] running on GCE node
I20260927 03:25:21.069739 13754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:21.070042 13754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:21.071180 13754 hybrid_clock.cc:648] HybridClock initialized: now 1790479521071168 us; error 29 us; skew 500 ppm
Sep 27 03:25:21 dist-test-slave-gw2v krb5kdc[13666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:21.073753 13754 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 03:25:21.074185 13754 webserver.cc:460] Webserver started at http://127.0.98.61:42317/ using document root <none> and password file <none>
I20260927 03:25:21.074401 13754 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:21.074452 13754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:21.074576 13754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:25:21.075701 13754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/data/instance:
uuid: "0d7ed8cac7684c1ba960005c36091c0f"
format_stamp: "Formatted at 2026-09-27 03:25:21 on dist-test-slave-gw2v"
server_key: "d1adff4cd5c463b28a59d6c735f573eb"
server_key_iv: "9238f74fd2317bea2fb0b8c6984741c2"
server_key_version: "encryptionkey@0"
I20260927 03:25:21.076062 13754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/wal/instance:
uuid: "0d7ed8cac7684c1ba960005c36091c0f"
format_stamp: "Formatted at 2026-09-27 03:25:21 on dist-test-slave-gw2v"
server_key: "d1adff4cd5c463b28a59d6c735f573eb"
server_key_iv: "9238f74fd2317bea2fb0b8c6984741c2"
server_key_version: "encryptionkey@0"
I20260927 03:25:21.077389 13754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:25:21.078147 13769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:21.078316 13754 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:25:21.078466 13754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/wal
uuid: "0d7ed8cac7684c1ba960005c36091c0f"
format_stamp: "Formatted at 2026-09-27 03:25:21 on dist-test-slave-gw2v"
server_key: "d1adff4cd5c463b28a59d6c735f573eb"
server_key_iv: "9238f74fd2317bea2fb0b8c6984741c2"
server_key_version: "encryptionkey@0"
I20260927 03:25:21.078531 13754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:21.080121 13747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } attempt: 2
I20260927 03:25:21.080541 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.080427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38385 (local address 127.0.0.1:54696)
0927 03:25:21.080481 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.080512 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 03:25:21.080863 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.080774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38385 (local address 127.0.0.1:54706)
0927 03:25:21.080819 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.080842 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 03:25:21.080935 13747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38385: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38385: connect: Connection refused (error 111)
I20260927 03:25:21.098691 13754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:21.101003 13754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:21.101142 13754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:21.105041 13754 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38385
I20260927 03:25:21.105070 13821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38385 every 8 connection(s)
I20260927 03:25:21.105412 13754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/data/info.pb
I20260927 03:25:21.106220 13822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:21.107872 13822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:25:21 dist-test-slave-gw2v krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:21.114913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 13754
I20260927 03:25:21.115012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-1/wal/instance
I20260927 03:25:21.115227 392 external_mini_cluster.cc:1428] Setting key fb87d566ffee4998a073fced1fdf59c1
I20260927 03:25:21.117128 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.108630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.0.98.62:43377)
0927 03:25:21.108700 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:21.108702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:21.108898 (+ 196us) server_negotiation.cc:407] Connection header received
0927 03:25:21.109379 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:21.109380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:21.109392 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:21.109439 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:21.110054 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.110591 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.111687 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.111897 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.111956 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.114077 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:21.114078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:21.114079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:21.114087 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:21.115635 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.116049 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.116049 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.116050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.116112 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.116438 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.116439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.116439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.116563 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:21.116624 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:21.116928 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 03:25:21.117034 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:25:21.117283 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.108574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43377 (local address 127.0.0.1:34756)
0927 03:25:21.108760 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.108797 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:21.109276 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:21.109516 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:21.109519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:21.109767 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:21.109968 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.109973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.110682 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.110683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.111607 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.111610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.111967 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.111969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.112087 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.112589 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:21.112606 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:21.113974 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:21.115719 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.115726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.115728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.115982 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.116191 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.116192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.116193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.116341 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.116624 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:21.116631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:21.116803 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 03:25:21.116897 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 03:25:21.117062 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260927 03:25:21.117837 13822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:25:21 dist-test-slave-gw2v krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 03:25:21.124869 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.118075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38385 (local address 127.0.0.1:54710)
0927 03:25:21.118112 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.118122 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:21.118153 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:21.118494 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:21.118495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:21.118658 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:21.118706 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.118707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.119564 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.119565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.120061 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.120062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.120489 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.120490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.120549 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.120973 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:21.120977 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:21.122222 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:21.123917 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.123918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.123918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.124079 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.124256 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.124257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.124257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.124310 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.124558 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:21.124558 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:21.124595 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 03:25:21.124654 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 03:25:21.124753 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 03:25:21.125008 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.118147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.0.98.61:38385)
0927 03:25:21.118372 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:21.118376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:21.118392 (+ 16us) server_negotiation.cc:407] Connection header received
0927 03:25:21.118408 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:21.118409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:21.118419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:21.118481 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:21.118769 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.119496 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.120125 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.120429 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.120484 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.122371 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:21.122378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:21.122379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:21.122384 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:21.123839 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.124147 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.124150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.124151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.124205 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.124359 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.124360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.124360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.124498 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:21.124574 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:21.124684 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 03:25:21.124822 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":198,"thread_start_us":88,"threads_started":1}
I20260927 03:25:21.125532 13822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:25:21.125837 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.125739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34615 (local address 127.0.0.1:42992)
0927 03:25:21.125770 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.125818 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260927 03:25:21.125945 13773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
I20260927 03:25:21.126330 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.126253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34615 (local address 127.0.0.1:43000)
0927 03:25:21.126289 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.126311 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 03:25:21.126420 13822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34615: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 03:25:21.137910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34615
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43377,127.0.98.61:38385,127.0.98.60:34615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:25:21.172096 13747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } attempt: 3
I20260927 03:25:21.175618 13822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } attempt: 1
I20260927 03:25:21.175946 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.175839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34615 (local address 127.0.0.1:43006)
0927 03:25:21.175885 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.175913 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
Sep 27 03:25:21 dist-test-slave-gw2v krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:25:21.176275 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.176194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34615 (local address 127.0.0.1:43018)
0927 03:25:21.176228 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.176250 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 03:25:21.176362 13822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34615: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
I20260927 03:25:21.178998 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.172608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54718 (local address 127.0.98.61:38385)
0927 03:25:21.172677 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:21.172679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:21.172688 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:25:21.172713 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:21.172714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:21.172725 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:21.172769 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:21.173123 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.173627 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.174293 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.174502 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.174558 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.176589 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:21.176590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:21.176591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:21.176596 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:21.177793 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.178171 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.178172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.178172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.178229 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.178509 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.178510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.178510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.178632 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:21.178697 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:21.178862 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 03:25:21.178944 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 03:25:21.179037 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.172517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38385 (local address 127.0.0.1:54718)
0927 03:25:21.172579 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.172591 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:21.172633 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:21.172785 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:21.172786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:21.172969 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:21.173035 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.173037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.173712 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.173713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.174212 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.174214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.174567 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.174569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.174643 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.175132 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:21.175137 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:21.176511 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:21.177869 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.177870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.177871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.178077 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.178359 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.178359 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.178360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.178423 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.178696 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:21.178696 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:21.178744 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 03:25:21.178849 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 03:25:21.178945 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 03:25:21.179404 13747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:25:21.179708 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.179606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34615 (local address 127.0.0.1:43020)
0927 03:25:21.179651 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.179685 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 03:25:21.180068 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.179973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34615 (local address 127.0.0.1:43034)
0927 03:25:21.180019 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.180048 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 03:25:21.180164 13747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34615: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
I20260927 03:25:21.211336 13747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } attempt: 1
I20260927 03:25:21.211865 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.211711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34615 (local address 127.0.0.1:43042)
0927 03:25:21.211782 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.211832 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 03:25:21.212275 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.212170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34615 (local address 127.0.0.1:43058)
0927 03:25:21.212225 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.212254 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 03:25:21.212384 13747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34615: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
W20260927 03:25:21.219133 13829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:21.219369 13829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:21.219401 13829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:21.220584 13829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:25:21.220628 13829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:21.220646 13829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:21.220661 13829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:25:21.220677 13829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:25:21.222193 13829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43377,127.0.98.61:38385,127.0.98.60:34615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:21.222424 13829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:21.222621 13829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:21.227200 13834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:21.227312 13829 server_base.cc:1034] running on GCE node
W20260927 03:25:21.227216 13837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:21.227406 13835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:21.227641 13829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:21.227900 13829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:21.229044 13829 hybrid_clock.cc:648] HybridClock initialized: now 1790479521229031 us; error 27 us; skew 500 ppm
Sep 27 03:25:21 dist-test-slave-gw2v krb5kdc[13666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:21.231442 13829 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 03:25:21.231854 13829 webserver.cc:460] Webserver started at http://127.0.98.60:44655/ using document root <none> and password file <none>
I20260927 03:25:21.232048 13829 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:21.232091 13829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:21.232195 13829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:25:21.233245 13829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/data/instance:
uuid: "532337ca376e404bb7681ad294f21e51"
format_stamp: "Formatted at 2026-09-27 03:25:21 on dist-test-slave-gw2v"
server_key: "8cd0e2015f03d323901343a1d368f43f"
server_key_iv: "b74b3cc70295cd159b5807f464755c77"
server_key_version: "encryptionkey@0"
I20260927 03:25:21.233564 13829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/wal/instance:
uuid: "532337ca376e404bb7681ad294f21e51"
format_stamp: "Formatted at 2026-09-27 03:25:21 on dist-test-slave-gw2v"
server_key: "8cd0e2015f03d323901343a1d368f43f"
server_key_iv: "b74b3cc70295cd159b5807f464755c77"
server_key_version: "encryptionkey@0"
I20260927 03:25:21.234793 13829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:25:21.235648 13844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:21.235858 13829 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:25:21.235949 13829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/wal
uuid: "532337ca376e404bb7681ad294f21e51"
format_stamp: "Formatted at 2026-09-27 03:25:21 on dist-test-slave-gw2v"
server_key: "8cd0e2015f03d323901343a1d368f43f"
server_key_iv: "b74b3cc70295cd159b5807f464755c77"
server_key_version: "encryptionkey@0"
I20260927 03:25:21.236012 13829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:21.244496 13822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } attempt: 2
I20260927 03:25:21.244906 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.244789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34615 (local address 127.0.0.1:43064)
0927 03:25:21.244845 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.244876 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 03:25:21.245218 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.245142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34615 (local address 127.0.0.1:43074)
0927 03:25:21.245178 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.245200 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 03:25:21.245296 13822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34615: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34615: connect: Connection refused (error 111)
I20260927 03:25:21.262727 13829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:21.265082 13829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:21.265225 13829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:21.269081 13829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34615
I20260927 03:25:21.269129 13896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34615 every 8 connection(s)
I20260927 03:25:21.269474 13829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/data/info.pb
I20260927 03:25:21.270366 13897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:21.271944 13897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:25:21.272611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 13829
I20260927 03:25:21.272696 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479369225731-392-0/minicluster-data/master-2/wal/instance
I20260927 03:25:21.272881 392 external_mini_cluster.cc:1428] Setting key a6fac82b7529f909ba39698bf942de15
Sep 27 03:25:21 dist-test-slave-gw2v krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 03:25:21 dist-test-slave-gw2v krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:25:21.282212 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.272694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.0.98.62:43377)
0927 03:25:21.272760 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:21.272762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:21.272964 (+ 202us) server_negotiation.cc:407] Connection header received
0927 03:25:21.273659 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:21.273661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:21.273674 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:21.273743 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:21.274263 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.274793 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.275985 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.276286 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.276379 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.278751 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:21.278752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:21.278752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:21.278776 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:21.280444 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.280909 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.280910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.280910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.280976 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.281426 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.281428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.281428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.281571 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:21.281648 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:21.282026 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 03:25:21.282116 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:25:21.282377 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.272655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43377 (local address 127.0.0.1:34766)
0927 03:25:21.272845 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.272887 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:21.273543 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:21.273768 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:21.273770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:21.274022 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:21.274176 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.274180 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.274886 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.274887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.275895 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.275897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.276473 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.276475 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.276631 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.277205 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:21.277213 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:21.278672 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:21.280528 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.280535 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.280536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.280810 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.281077 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.281078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.281079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.281237 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.281656 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:21.281664 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:21.281832 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 03:25:21.281941 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 03:25:21.282107 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":109,"thread_start_us":78,"threads_started":1}
I20260927 03:25:21.282377 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.273280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.0.98.60:34615)
0927 03:25:21.273424 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:21.273429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:21.273466 (+ 37us) server_negotiation.cc:407] Connection header received
0927 03:25:21.273553 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:21.273561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:21.273584 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:21.273664 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:21.274379 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.275482 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.276486 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.276924 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.276977 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.279265 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:21.279274 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:21.279274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:21.279281 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:21.281023 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.281433 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.281436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.281436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.281490 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.281692 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.281693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.281693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.281874 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:21.282073 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:21.282149 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 03:25:21.282213 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260927 03:25:21.283288 13897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:25:21 dist-test-slave-gw2v krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:25:21.287580 13747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } attempt: 2
Sep 27 03:25:21 dist-test-slave-gw2v krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:21.290953 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.283541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38385 (local address 127.0.0.1:54734)
0927 03:25:21.283582 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.283590 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:21.283618 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:21.283727 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:21.283728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:21.283882 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:21.283947 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.283949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.284805 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.284806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.285523 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.285525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.285911 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.285912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.285977 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.286392 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:21.286397 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:21.287881 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:21.289826 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.289827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.289828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.290031 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.290233 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.290233 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.290234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.290288 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.290623 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:21.290624 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:21.290675 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 03:25:21.290731 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:25:21.290849 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 03:25:21.290977 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.283590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.0.98.61:38385)
0927 03:25:21.283640 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:21.283642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:21.283649 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:25:21.283671 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:21.283672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:21.283682 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:21.283726 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:21.284020 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.284704 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.285624 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.285819 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.285895 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.287990 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:21.287992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:21.287992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:21.288000 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:21.289730 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.290109 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.290110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.290110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.290163 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.290407 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.290408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.290409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.290547 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:21.290617 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:21.290777 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 03:25:21.290897 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 27 03:25:21 dist-test-slave-gw2v krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:25:21.291826 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.283484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.0.98.62:43377)
0927 03:25:21.283527 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:21.283529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:21.283577 (+ 48us) server_negotiation.cc:407] Connection header received
0927 03:25:21.283830 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:21.283831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:21.283839 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:21.283884 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:21.284333 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.284945 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.285889 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.286140 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.286204 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.288534 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:21.288535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:21.288536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:21.288543 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:21.290013 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.290377 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.290378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.290378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.290430 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.290593 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.290594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.290594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.290737 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:21.291392 (+ 655us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:21.291665 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 03:25:21.291744 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:25:21.291812 13897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:25:21.295689 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.287837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34615 (local address 127.0.0.1:43092)
0927 03:25:21.287896 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.287906 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:21.287944 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:21.288061 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:21.288061 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:21.288217 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:21.288260 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.288262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.289135 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.289136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.289706 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.289708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.290068 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.290069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.290124 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.290659 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:21.290664 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:21.292066 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:21.294651 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.294652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.294653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.294900 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.295084 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.295085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.295085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.295147 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.295412 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:21.295413 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:21.295459 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 03:25:21.295506 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:25:21.295593 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 03:25:21.296160 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.287887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43092 (local address 127.0.98.60:34615)
0927 03:25:21.287922 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:21.287924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:21.287931 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:25:21.287997 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:21.287998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:21.288008 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:21.288048 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:21.288338 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.289032 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.289785 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.290007 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.290061 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.292132 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:21.292133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:21.292133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:21.292140 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:21.294589 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.294972 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.294973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.294974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.295026 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.295207 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.295208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.295209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.295331 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:21.295972 (+ 641us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:21.296012 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 03:25:21.296081 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 03:25:21 dist-test-slave-gw2v krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 03:25:21 dist-test-slave-gw2v krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:25:21.297571 13747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7: Bootstrap starting.
I20260927 03:25:21.298313 13747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:21.298763 13747 log.cc:824] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:21.299643 13747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7: No bootstrap required, opened a new log
I20260927 03:25:21.300496 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.292048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34615 (local address 127.0.0.1:43100)
0927 03:25:21.292360 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.292377 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:21.292414 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:21.292728 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:21.292729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:21.292873 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:21.292908 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.292910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.293989 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.293990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.294472 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.294474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.295273 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.295274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.295319 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.295723 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:21.295727 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:21.296992 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:21.298659 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.298660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.298660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.299692 (+ 1032us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.299865 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.299865 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.299866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.299913 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.300210 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:21.300210 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:21.300246 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 03:25:21.300315 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:25:21.300409 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297}
I20260927 03:25:21.300748 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.292224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43100 (local address 127.0.98.60:34615)
0927 03:25:21.292607 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:21.292609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:21.292616 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:25:21.292631 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:21.292632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:21.292640 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:21.292696 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:21.293010 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.293915 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.294885 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.295177 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.295235 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.297076 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:21.297077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:21.297077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:21.297083 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:21.298596 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.299760 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.299760 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.299761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.299810 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.299968 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.299968 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.299968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.300134 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:21.300191 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:21.300593 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 03:25:21.300663 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":96,"threads_started":1}
I20260927 03:25:21.301018 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.292561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.0.98.61:38385)
0927 03:25:21.292598 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:21.292599 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:21.292606 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:25:21.292626 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:21.292627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:21.292636 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:21.292692 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:21.293211 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.293654 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.294571 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.294864 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.294932 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.297899 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:21.297900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:21.297901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:21.297907 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:21.299536 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.300071 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.300072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.300072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.300125 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.300321 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.300322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.300322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.300449 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:21.300623 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:21.300856 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 03:25:21.300931 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:25:21.302465 13897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51: Bootstrap starting.
I20260927 03:25:21.303244 13897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:21.303566 13897 log.cc:824] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:21.304388 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:25:21.305240 13747 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } }
I20260927 03:25:21.305531 13747 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:21.305992 13897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51: No bootstrap required, opened a new log
I20260927 03:25:21.306080 13747 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5e602383804aeea278ca8aceb106d7, State: Initialized, Role: FOLLOWER
I20260927 03:25:21.306331 13747 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } }
I20260927 03:25:21.306772 13747 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:25:21.308696 13897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } }
I20260927 03:25:21.308818 13897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:21.308840 13897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 532337ca376e404bb7681ad294f21e51, State: Initialized, Role: FOLLOWER
I20260927 03:25:21.309020 13897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } }
I20260927 03:25:21.309355 13897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:25:21.306926 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } } }
I20260927 03:25:21.309567 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:25:21.309365 13913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } } }
I20260927 03:25:21.310700 13913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:25:21.311201 13924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:25:21.311319 13924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 03:25:21.312973 13936 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:25:21.313124 13936 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:25:21.313876 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.305884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.0.98.60:34615)
0927 03:25:21.305921 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:21.305923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:21.306049 (+ 126us) server_negotiation.cc:407] Connection header received
0927 03:25:21.306674 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:21.306675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:21.306686 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:21.306735 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:21.307823 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.308436 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.309212 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.309422 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.309476 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.310605 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:21.310607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:21.310607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:21.310614 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:21.312492 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.312963 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.312964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.312965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.313075 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.313270 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.313271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.313271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.313403 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:21.313500 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:21.313696 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 03:25:21.313779 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:25:21.311946 13679 master_runner.cc:186] Error getting master registration for 127.0.98.61:38385: OK, instance_id { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" instance_seqno: 1790479521103919 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:25:21.316380 13679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:25:21.316874 13829 master_runner.cc:186] Error getting master registration for 127.0.98.61:38385: OK, instance_id { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" instance_seqno: 1790479521103919 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:25:21.318709 13829 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:25:21.323082 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.306346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34794 (local address 127.0.98.62:43377)
0927 03:25:21.306382 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:21.306384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:21.312264 (+ 5880us) server_negotiation.cc:407] Connection header received
0927 03:25:21.313301 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:21.313302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:21.313313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:21.313341 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:21.313921 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.314654 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.315637 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.315917 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.315980 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.320178 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:21.320179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:21.320180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:21.320187 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:21.321715 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.322128 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.322130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.322130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.322188 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.322374 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.322375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.322376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.322507 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:21.322608 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:21.322892 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 03:25:21.322983 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:25:21.323849 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.306525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.0.98.61:38385)
0927 03:25:21.306638 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:21.306640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:21.312359 (+ 5719us) server_negotiation.cc:407] Connection header received
0927 03:25:21.312508 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:21.312509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:21.312538 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:21.312578 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:21.315690 (+ 3112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.316404 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.317399 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.317678 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.317746 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.320316 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:21.320317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:21.320318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:21.320327 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:21.322348 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.322719 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.322721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.322721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.322785 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.323014 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.323015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.323015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.323141 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:21.323224 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:21.323664 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 03:25:21.323754 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 03:25:21.336421 13822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } attempt: 3
Sep 27 03:25:21 dist-test-slave-gw2v krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:25:21.345680 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.336696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34615 (local address 127.0.0.1:43128)
0927 03:25:21.337821 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:21.337835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:21.337894 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:21.338059 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:21.338060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:21.338252 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:21.338309 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.338312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.339291 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.339293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.340165 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:21.340168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.340616 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.340617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:21.340676 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.341207 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:21.341212 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:21.342699 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:21.344630 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.344632 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.344632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.344854 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.345051 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:21.345052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:21.345052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:21.345115 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:21.345357 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:21.345358 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:21.345411 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 03:25:21.345477 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:25:21.345576 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 03:25:21.345680 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:21.336749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.0.98.60:34615)
0927 03:25:21.336795 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:21.336796 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:21.337879 (+ 1083us) server_negotiation.cc:407] Connection header received
0927 03:25:21.337964 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:21.337965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:21.337976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:21.338050 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:21.338380 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.339213 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.340240 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:21.340536 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:21.340614 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:21.342782 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:21.342784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:21.342784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:21.342792 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:21.344550 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.344932 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.344933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.344933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.344992 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:21.345170 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:21.345171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:21.345171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:21.345299 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:21.345363 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:21.345506 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 03:25:21.345584 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:25:21.347731 13822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f: Bootstrap starting.
I20260927 03:25:21.348483 13822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:21.348821 13822 log.cc:824] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:21.349664 13822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f: No bootstrap required, opened a new log
I20260927 03:25:21.352167 13822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } }
I20260927 03:25:21.352309 13822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:21.352339 13822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7ed8cac7684c1ba960005c36091c0f, State: Initialized, Role: FOLLOWER
I20260927 03:25:21.352495 13822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } }
I20260927 03:25:21.352841 13822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:25:21.352813 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } } }
I20260927 03:25:21.353102 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:25:21.354806 13950 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:25:21.354921 13950 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:25:21.356555 13754 master_runner.cc:263] No leader master found from master 0d7ed8cac7684c1ba960005c36091c0f
I20260927 03:25:21.356598 13754 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 03:25:21.443602 13700 tablet.cc:2369] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:25:21.458997 13912 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:25:21.459192 13912 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } }
I20260927 03:25:21.459391 13912 leader_election.cc:288] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7ed8cac7684c1ba960005c36091c0f (127.0.98.61:38385), 532337ca376e404bb7681ad294f21e51 (127.0.98.60:34615)
I20260927 03:25:21.459636 13796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba5e602383804aeea278ca8aceb106d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7ed8cac7684c1ba960005c36091c0f" is_pre_election: true
I20260927 03:25:21.459781 13796 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba5e602383804aeea278ca8aceb106d7 in term 0.
I20260927 03:25:21.459798 13871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba5e602383804aeea278ca8aceb106d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "532337ca376e404bb7681ad294f21e51" is_pre_election: true
I20260927 03:25:21.459934 13871 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba5e602383804aeea278ca8aceb106d7 in term 0.
I20260927 03:25:21.460088 13695 leader_election.cc:302] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7ed8cac7684c1ba960005c36091c0f, ba5e602383804aeea278ca8aceb106d7; no voters:
I20260927 03:25:21.460211 13912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:25:21.460253 13912 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:25:21.460278 13912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:21.461234 13912 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } }
I20260927 03:25:21.461337 13912 leader_election.cc:288] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7ed8cac7684c1ba960005c36091c0f (127.0.98.61:38385), 532337ca376e404bb7681ad294f21e51 (127.0.98.60:34615)
I20260927 03:25:21.461521 13871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba5e602383804aeea278ca8aceb106d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "532337ca376e404bb7681ad294f21e51"
I20260927 03:25:21.461545 13796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba5e602383804aeea278ca8aceb106d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7ed8cac7684c1ba960005c36091c0f"
I20260927 03:25:21.461606 13796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:21.461598 13871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:21.462366 13796 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba5e602383804aeea278ca8aceb106d7 in term 1.
I20260927 03:25:21.462365 13871 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba5e602383804aeea278ca8aceb106d7 in term 1.
I20260927 03:25:21.462551 13698 leader_election.cc:302] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 532337ca376e404bb7681ad294f21e51, ba5e602383804aeea278ca8aceb106d7; no voters:
I20260927 03:25:21.462656 13912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:25:21.462769 13912 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [term 1 LEADER]: Becoming Leader. State: Replica: ba5e602383804aeea278ca8aceb106d7, State: Running, Role: LEADER
I20260927 03:25:21.462877 13912 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } }
I20260927 03:25:21.463253 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba5e602383804aeea278ca8aceb106d7. Latest consensus state: current_term: 1 leader_uuid: "ba5e602383804aeea278ca8aceb106d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } } }
I20260927 03:25:21.463351 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:21.463752 13954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:25:21.463915 13954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:25:21.465130 13871 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51 [term 1 FOLLOWER]: Refusing update from remote peer ba5e602383804aeea278ca8aceb106d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:25:21.465194 13796 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f [term 1 FOLLOWER]: Refusing update from remote peer ba5e602383804aeea278ca8aceb106d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:25:21.465487 13952 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:25:21.465572 13912 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:25:21.466737 13913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba5e602383804aeea278ca8aceb106d7. Latest consensus state: current_term: 1 leader_uuid: "ba5e602383804aeea278ca8aceb106d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } } }
I20260927 03:25:21.466853 13913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:25:21.466787 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba5e602383804aeea278ca8aceb106d7. Latest consensus state: current_term: 1 leader_uuid: "ba5e602383804aeea278ca8aceb106d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } } }
I20260927 03:25:21.466915 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:25:21.467661 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba5e602383804aeea278ca8aceb106d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } } }
I20260927 03:25:21.467744 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:21.468156 13913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba5e602383804aeea278ca8aceb106d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } } }
I20260927 03:25:21.468230 13913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:25:21.468309 13956 mvcc.cc:205] Tried to move back new op lower bound from 7333804119919448064 to 7333804119912390656. Current Snapshot: MvccSnapshot[applied={T|T < 7333804119919448064}]
I20260927 03:25:21.468780 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba5e602383804aeea278ca8aceb106d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } } }
I20260927 03:25:21.468850 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7 [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:21.469239 13957 mvcc.cc:205] Tried to move back new op lower bound from 7333804119919448064 to 7333804119912390656. Current Snapshot: MvccSnapshot[applied={T|T < 7333804119919448064}]
I20260927 03:25:21.469319 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba5e602383804aeea278ca8aceb106d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e602383804aeea278ca8aceb106d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } peers { permanent_uuid: "0d7ed8cac7684c1ba960005c36091c0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38385 } } peers { permanent_uuid: "532337ca376e404bb7681ad294f21e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34615 } } }
I20260927 03:25:21.469384 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:25:21.471204 13954 catalog_manager.cc:1347] Generated new cluster ID: 8c26dfb48fee4ea5a4fbfb7d2ea1e456
I20260927 03:25:21.471262 13954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:25:21.493634 13954 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:25:21.494231 13954 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:25:21.499042 13954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba5e602383804aeea278ca8aceb106d7: Generated new TSK 0
I20260927 03:25:21.499240 13954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:25:22.313597 13936 catalog_manager.cc:1259] Loaded cluster ID: 8c26dfb48fee4ea5a4fbfb7d2ea1e456
I20260927 03:25:22.313689 13936 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51: loading cluster ID for follower catalog manager: success
I20260927 03:25:22.314491 13936 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51: acquiring CA information for follower catalog manager: success
I20260927 03:25:22.314805 13936 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 532337ca376e404bb7681ad294f21e51: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:25:22.355419 13950 catalog_manager.cc:1259] Loaded cluster ID: 8c26dfb48fee4ea5a4fbfb7d2ea1e456
I20260927 03:25:22.355501 13950 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f: loading cluster ID for follower catalog manager: success
I20260927 03:25:22.356294 13950 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f: acquiring CA information for follower catalog manager: success
I20260927 03:25:22.356581 13950 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d7ed8cac7684c1ba960005c36091c0f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:25:22.563359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 13679
I20260927 03:25:22.567909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 13754
I20260927 03:25:22.572319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 13829
2026-09-27T03:25:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2835 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:25:22 dist-test-slave-gw2v krb5kdc[13969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:25:22 dist-test-slave-gw2v krb5kdc[13969](info): set up 2 sockets
Sep 27 03:25:22 dist-test-slave-gw2v krb5kdc[13969](info): commencing operation
krb5kdc: starting...
W20260927 03:25:23.522614 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:25:23 dist-test-slave-gw2v krb5kdc[13969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:25:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:25:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:25:23.641232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32779
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:23.723282 13982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:23.723501 13982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:23.723524 13982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:23.724720 13982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:25:23.724768 13982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:23.724782 13982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:23.724793 13982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:25:23.724803 13982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:25:23.726282 13982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:23.726495 13982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:23.726687 13982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:23.731444 13988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:23.731417 13987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:23.731431 13990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:23.731722 13982 server_base.cc:1034] running on GCE node
I20260927 03:25:23.732003 13982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:23.732309 13982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:23.733458 13982 hybrid_clock.cc:648] HybridClock initialized: now 1790479523733426 us; error 50 us; skew 500 ppm
Sep 27 03:25:23 dist-test-slave-gw2v krb5kdc[13969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:23.735936 13982 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:25:23.736340 13982 webserver.cc:460] Webserver started at http://127.0.98.62:33409/ using document root <none> and password file <none>
I20260927 03:25:23.736526 13982 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:23.736565 13982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:23.736660 13982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:25:23.737704 13982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "db9a4f245ffe45589efa5d502a73a982"
format_stamp: "Formatted at 2026-09-27 03:25:23 on dist-test-slave-gw2v"
server_key: "ac97e2c3b96087afa5d1a7d69822b292"
server_key_iv: "968118b1207cc5e3d3851e60e4164bf2"
server_key_version: "encryptionkey@0"
I20260927 03:25:23.737991 13982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "db9a4f245ffe45589efa5d502a73a982"
format_stamp: "Formatted at 2026-09-27 03:25:23 on dist-test-slave-gw2v"
server_key: "ac97e2c3b96087afa5d1a7d69822b292"
server_key_iv: "968118b1207cc5e3d3851e60e4164bf2"
server_key_version: "encryptionkey@0"
I20260927 03:25:23.739323 13982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:25:23.740012 13997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:23.740178 13982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:25:23.740262 13982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "db9a4f245ffe45589efa5d502a73a982"
format_stamp: "Formatted at 2026-09-27 03:25:23 on dist-test-slave-gw2v"
server_key: "ac97e2c3b96087afa5d1a7d69822b292"
server_key_iv: "968118b1207cc5e3d3851e60e4164bf2"
server_key_version: "encryptionkey@0"
I20260927 03:25:23.740322 13982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:23.753041 13982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:23.755357 13982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:23.755501 13982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:23.759254 13982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32779
I20260927 03:25:23.759281 14049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32779 every 8 connection(s)
I20260927 03:25:23.759619 13982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:25:23.760442 14050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:23.762630 14050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982: Bootstrap starting.
I20260927 03:25:23.763209 14050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:23.763433 14050 log.cc:824] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:23.764070 14050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982: No bootstrap required, opened a new log
I20260927 03:25:23.765604 14050 raft_consensus.cc:348] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9a4f245ffe45589efa5d502a73a982" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260927 03:25:23.765763 14050 raft_consensus.cc:374] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:23.765812 14050 raft_consensus.cc:729] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9a4f245ffe45589efa5d502a73a982, State: Initialized, Role: FOLLOWER
I20260927 03:25:23.765957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 13982
I20260927 03:25:23.765947 14050 consensus_queue.cc:260] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9a4f245ffe45589efa5d502a73a982" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260927 03:25:23.766019 14050 raft_consensus.cc:388] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:25:23.766050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:25:23.766058 14050 raft_consensus.cc:482] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:25:23.766103 14050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:23.766228 392 external_mini_cluster.cc:1428] Setting key 86bdc8e9934aad858ffb8dfcb20898b8
I20260927 03:25:23.766721 14050 raft_consensus.cc:504] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9a4f245ffe45589efa5d502a73a982" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260927 03:25:23.766882 14050 leader_election.cc:302] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db9a4f245ffe45589efa5d502a73a982; no voters:
I20260927 03:25:23.767035 14050 leader_election.cc:288] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:25:23.767071 14055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:25:23.767261 14055 raft_consensus.cc:686] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [term 1 LEADER]: Becoming Leader. State: Replica: db9a4f245ffe45589efa5d502a73a982, State: Running, Role: LEADER
I20260927 03:25:23.767269 14050 sys_catalog.cc:564] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:25:23.767402 14055 consensus_queue.cc:237] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9a4f245ffe45589efa5d502a73a982" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260927 03:25:23.767686 14055 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db9a4f245ffe45589efa5d502a73a982. Latest consensus state: current_term: 1 leader_uuid: "db9a4f245ffe45589efa5d502a73a982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9a4f245ffe45589efa5d502a73a982" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } } }
I20260927 03:25:23.767745 14055 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:23.768049 14061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:25:23.768558 14061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:25:23.768746 14056 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db9a4f245ffe45589efa5d502a73a982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9a4f245ffe45589efa5d502a73a982" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } } }
I20260927 03:25:23.769658 14056 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982 [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:23.770051 14072 mvcc.cc:205] Tried to move back new op lower bound from 7333804129359417344 to 7333804129351598080. Current Snapshot: MvccSnapshot[applied={T|T < 7333804129359417344}]
I20260927 03:25:23.770532 14061 catalog_manager.cc:1347] Generated new cluster ID: 045bdc558a71460a86140d6dc71420cc
I20260927 03:25:23.770584 14061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:25:23 dist-test-slave-gw2v krb5kdc[13969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:23.775919 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:23.766644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.0.98.62:32779)
0927 03:25:23.766838 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:23.766849 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:23.766886 (+ 37us) server_negotiation.cc:407] Connection header received
0927 03:25:23.767002 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:23.767008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:23.767065 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:23.767213 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:23.767951 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:23.769284 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:23.770286 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:23.770868 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:23.770972 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:23.773074 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:23.773077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:23.773078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:23.773090 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:23.774559 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:23.774893 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:23.774897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:23.774897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:23.774991 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:23.775162 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:23.775163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:23.775163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:23.775404 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:23.775543 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:23.775634 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:25:23.775766 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260927 03:25:23.776759 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:25:23.782183 14061 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:25:23.783002 14061 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:25:23.800940 14061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db9a4f245ffe45589efa5d502a73a982: Generated new TSK 0
I20260927 03:25:23.801191 14061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:25:23.830761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 13982
2026-09-27T03:25:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1257 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:25:23 dist-test-slave-gw2v krb5kdc[14077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:25:23 dist-test-slave-gw2v krb5kdc[14077](info): set up 2 sockets
Sep 27 03:25:23 dist-test-slave-gw2v krb5kdc[14077](info): commencing operation
krb5kdc: starting...
W20260927 03:25:24.782725 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:25:24 dist-test-slave-gw2v krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:25:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:25:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:25:24.901808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35283
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:24.981530 14090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:24.981747 14090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:24.981768 14090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:24.982924 14090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:25:24.982959 14090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:24.982972 14090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:24.982983 14090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:25:24.982993 14090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:25:24.984443 14090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:24.984647 14090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:24.984823 14090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:24.989575 14098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:24.989656 14090 server_base.cc:1034] running on GCE node
W20260927 03:25:24.989589 14096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:24.989630 14095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:24.990022 14090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:24.990314 14090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:24.991461 14090 hybrid_clock.cc:648] HybridClock initialized: now 1790479524991447 us; error 28 us; skew 500 ppm
Sep 27 03:25:24 dist-test-slave-gw2v krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:24.993858 14090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:25:24.994272 14090 webserver.cc:460] Webserver started at http://127.0.98.62:36155/ using document root <none> and password file <none>
I20260927 03:25:24.994473 14090 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:24.994513 14090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:24.994612 14090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:25:24.995705 14090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "4d407a2dc4614e3abadc780570705a9e"
format_stamp: "Formatted at 2026-09-27 03:25:24 on dist-test-slave-gw2v"
server_key: "990a8319f06426d659cbc8fc9716f82d"
server_key_iv: "e2f0e5d6bbf8b93f76cf5a1db62fd013"
server_key_version: "encryptionkey@0"
I20260927 03:25:24.996017 14090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d407a2dc4614e3abadc780570705a9e"
format_stamp: "Formatted at 2026-09-27 03:25:24 on dist-test-slave-gw2v"
server_key: "990a8319f06426d659cbc8fc9716f82d"
server_key_iv: "e2f0e5d6bbf8b93f76cf5a1db62fd013"
server_key_version: "encryptionkey@0"
I20260927 03:25:24.997339 14090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:25:24.998021 14105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:24.998211 14090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:25:24.998301 14090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "4d407a2dc4614e3abadc780570705a9e"
format_stamp: "Formatted at 2026-09-27 03:25:24 on dist-test-slave-gw2v"
server_key: "990a8319f06426d659cbc8fc9716f82d"
server_key_iv: "e2f0e5d6bbf8b93f76cf5a1db62fd013"
server_key_version: "encryptionkey@0"
I20260927 03:25:24.998358 14090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:25.008349 14090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:25.010731 14090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:25.010903 14090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:25.014695 14090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35283
I20260927 03:25:25.014753 14157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35283 every 8 connection(s)
I20260927 03:25:25.015093 14090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:25:25.016083 14158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:25.016531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 14090
I20260927 03:25:25.016618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:25:25.016780 392 external_mini_cluster.cc:1428] Setting key b320a933da4e0cfc73e1e2d6bd3cd207
I20260927 03:25:25.018882 14158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: Bootstrap starting.
I20260927 03:25:25.019536 14158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:25.019794 14158 log.cc:824] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:25.020402 14158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: No bootstrap required, opened a new log
I20260927 03:25:25.022133 14158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260927 03:25:25.022272 14158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:25.022305 14158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d407a2dc4614e3abadc780570705a9e, State: Initialized, Role: FOLLOWER
I20260927 03:25:25.022425 14158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260927 03:25:25.022487 14158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:25:25.022532 14158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:25:25.022578 14158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 0 FOLLOWER]: Advancing to term 1
Sep 27 03:25:25 dist-test-slave-gw2v krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:25.023140 14158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260927 03:25:25.023232 14158 leader_election.cc:302] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d407a2dc4614e3abadc780570705a9e; no voters:
I20260927 03:25:25.023383 14158 leader_election.cc:288] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:25:25.023425 14163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:25:25.023530 14163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 1 LEADER]: Becoming Leader. State: Replica: 4d407a2dc4614e3abadc780570705a9e, State: Running, Role: LEADER
I20260927 03:25:25.023663 14158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:25:25.023685 14163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260927 03:25:25.024065 14163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d407a2dc4614e3abadc780570705a9e. Latest consensus state: current_term: 1 leader_uuid: "4d407a2dc4614e3abadc780570705a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } } }
I20260927 03:25:25.024192 14163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:25.024117 14164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d407a2dc4614e3abadc780570705a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } } }
I20260927 03:25:25.024564 14164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:25.024488 14172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:25:25.025224 14172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:25:25.026889 14172 catalog_manager.cc:1347] Generated new cluster ID: ac2d5f874b464eae85c6c6300fe44183
I20260927 03:25:25.026939 14172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:25:25.027406 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.017406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.0.98.62:35283)
0927 03:25:25.017555 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:25.017563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:25.017588 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:25:25.017983 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:25.017989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:25.018035 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:25.018150 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:25.018965 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.020017 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.020792 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.021231 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.021330 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.023672 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:25.023676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:25.023677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:25.023690 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:25.025722 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:25.026090 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:25.026094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:25.026094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:25.026188 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:25.026360 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:25.026361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:25.026362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:25.026646 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:25.026792 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:25.027088 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 03:25:25.027223 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260927 03:25:25.035084 14172 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:25:25.035609 14172 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:25:25.039855 14172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: Generated new TSK 0
I20260927 03:25:25.040061 14172 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:25:25.073263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35283
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:25.169651 14184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:25.169866 14184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:25.169889 14184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:25.171039 14184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:25.171077 14184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:25.171103 14184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:25:25.172523 14184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:25.172708 14184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:25.172883 14184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:25:25.177747 14184 server_base.cc:1034] running on GCE node
W20260927 03:25:25.177863 14189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:25.177821 14190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:25.177745 14192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:25.178221 14184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:25.178462 14184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:25.179621 14184 hybrid_clock.cc:648] HybridClock initialized: now 1790479525179587 us; error 50 us; skew 500 ppm
Sep 27 03:25:25 dist-test-slave-gw2v krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:25.182155 14184 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:25:25.182610 14184 webserver.cc:460] Webserver started at http://127.0.98.1:36891/ using document root <none> and password file <none>
I20260927 03:25:25.182838 14184 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:25.182898 14184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:25.183107 14184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:25:25.184167 14184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "d72e99c6079842e6a6ba2f37bdc136f9"
format_stamp: "Formatted at 2026-09-27 03:25:25 on dist-test-slave-gw2v"
server_key: "f748b207047a4e5a8dc2be9c136fdfbf"
server_key_iv: "e48c9b07ca9fe6f3cb95b174e92056b7"
server_key_version: "encryptionkey@0"
I20260927 03:25:25.184515 14184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d72e99c6079842e6a6ba2f37bdc136f9"
format_stamp: "Formatted at 2026-09-27 03:25:25 on dist-test-slave-gw2v"
server_key: "f748b207047a4e5a8dc2be9c136fdfbf"
server_key_iv: "e48c9b07ca9fe6f3cb95b174e92056b7"
server_key_version: "encryptionkey@0"
I20260927 03:25:25.185807 14184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:25:25.186638 14199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:25.186841 14184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:25:25.187001 14184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "d72e99c6079842e6a6ba2f37bdc136f9"
format_stamp: "Formatted at 2026-09-27 03:25:25 on dist-test-slave-gw2v"
server_key: "f748b207047a4e5a8dc2be9c136fdfbf"
server_key_iv: "e48c9b07ca9fe6f3cb95b174e92056b7"
server_key_version: "encryptionkey@0"
I20260927 03:25:25.187075 14184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:25.194715 14184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:25.196823 14184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:25.196972 14184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:25.197178 14184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:25.197472 14184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:25.197504 14184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:25.197531 14184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:25.197556 14184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:25.203327 14184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46281
I20260927 03:25:25.203363 14312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46281 every 8 connection(s)
I20260927 03:25:25.203699 14184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:25:25.208673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 14184
I20260927 03:25:25.208776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:25:25.208981 392 external_mini_cluster.cc:1428] Setting key dd62982d2e506470a7e894b63945f595
Sep 27 03:25:25 dist-test-slave-gw2v krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:25.213490 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.204397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54407 (local address 127.0.98.62:35283)
0927 03:25:25.204484 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:25.204491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:25.204665 (+ 174us) server_negotiation.cc:407] Connection header received
0927 03:25:25.205175 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:25.205176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:25.205186 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:25.205253 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:25.206263 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.206780 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.207940 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.208154 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.208213 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.210606 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:25.210607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:25.210607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:25.210613 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:25.212106 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:25.212521 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:25.212522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:25.212523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:25.212582 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:25.212804 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:25.212805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:25.212806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:25.212925 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:25.212981 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:25.213295 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 03:25:25.213391 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 03:25:25.213589 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.204370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:54407)
0927 03:25:25.204532 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:25.204580 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:25.205067 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:25.205330 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:25.205333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:25.205901 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:25.206152 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.206158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.206984 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.206984 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.207800 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.207802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.208223 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.208224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.208322 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.208940 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:25.208953 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:25.210497 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:25.212190 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:25.212197 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:25.212199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:25.212429 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:25.212639 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:25.212639 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:25.212640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:25.212749 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:25.212982 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:25.212989 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:25.213129 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 03:25:25.213240 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 03:25:25.213401 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260927 03:25:25.214147 14313 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35283
I20260927 03:25:25.214257 14313 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:25.214453 14313 heartbeater.cc:511] Master 127.0.98.62:35283 requested a full tablet report, sending...
I20260927 03:25:25.215051 14122 ts_manager.cc:194] Registered new tserver with Master: d72e99c6079842e6a6ba2f37bdc136f9 (127.0.98.1:46281)
I20260927 03:25:25.215660 14122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54407
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:25:25.234463 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35283
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:25.316612 14318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:25.316843 14318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:25.316885 14318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:25.318059 14318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:25.318102 14318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:25.318138 14318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:25:25.319689 14318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:25.319949 14318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:25.320164 14318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:25.324997 14323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:25.325039 14326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:25.325040 14324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:25.325251 14318 server_base.cc:1034] running on GCE node
I20260927 03:25:25.325444 14318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:25.325668 14318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:25.326810 14318 hybrid_clock.cc:648] HybridClock initialized: now 1790479525326798 us; error 29 us; skew 500 ppm
Sep 27 03:25:25 dist-test-slave-gw2v krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:25.329545 14318 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:25:25.330041 14318 webserver.cc:460] Webserver started at http://127.0.98.2:36633/ using document root <none> and password file <none>
I20260927 03:25:25.330268 14318 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:25.330320 14318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:25.330447 14318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:25:25.331573 14318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "2ba88939b2cb4e60b90af9e2275f57b0"
format_stamp: "Formatted at 2026-09-27 03:25:25 on dist-test-slave-gw2v"
server_key: "53b71e254aed891b4cbc08b293190836"
server_key_iv: "2af91cc0120951223f351425fdfa9776"
server_key_version: "encryptionkey@0"
I20260927 03:25:25.331918 14318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2ba88939b2cb4e60b90af9e2275f57b0"
format_stamp: "Formatted at 2026-09-27 03:25:25 on dist-test-slave-gw2v"
server_key: "53b71e254aed891b4cbc08b293190836"
server_key_iv: "2af91cc0120951223f351425fdfa9776"
server_key_version: "encryptionkey@0"
I20260927 03:25:25.333281 14318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:25:25.334107 14333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:25.334312 14318 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:25:25.334405 14318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "2ba88939b2cb4e60b90af9e2275f57b0"
format_stamp: "Formatted at 2026-09-27 03:25:25 on dist-test-slave-gw2v"
server_key: "53b71e254aed891b4cbc08b293190836"
server_key_iv: "2af91cc0120951223f351425fdfa9776"
server_key_version: "encryptionkey@0"
I20260927 03:25:25.334474 14318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:25.344271 14318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:25.346433 14318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:25.346573 14318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:25.346792 14318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:25.347187 14318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:25.347222 14318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:25.347252 14318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:25.347270 14318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:25.352694 14318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46143
I20260927 03:25:25.352756 14446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46143 every 8 connection(s)
I20260927 03:25:25.353070 14318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:25:25 dist-test-slave-gw2v krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:25.359599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 14318
I20260927 03:25:25.359681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:25:25.359864 392 external_mini_cluster.cc:1428] Setting key 799d340f60c7a33166962298b933221c
I20260927 03:25:25.362906 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.353830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45465 (local address 127.0.98.62:35283)
0927 03:25:25.353925 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:25.353927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:25.354107 (+ 180us) server_negotiation.cc:407] Connection header received
0927 03:25:25.354562 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:25.354563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:25.354573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:25.354634 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:25.355587 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.356131 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.357190 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.357406 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.357480 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.359850 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:25.359852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:25.359853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:25.359861 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:25.361338 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:25.361837 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:25.361838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:25.361839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:25.361908 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:25.362240 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:25.362241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:25.362241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:25.362354 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:25.362406 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:25.362729 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 03:25:25.362797 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:25:25.363145 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.353788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:45465)
0927 03:25:25.353943 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:25.353985 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:25.354478 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:25.354693 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:25.354696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:25.355287 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:25.355492 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.355497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.356239 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.356240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.357074 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.357077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.357510 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.357511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.357636 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.358284 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:25.358292 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:25.359752 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:25.361437 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:25.361445 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:25.361448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:25.361741 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:25.361977 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:25.361977 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:25.361978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:25.362176 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:25.362411 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:25.362416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:25.362589 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 03:25:25.362705 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 03:25:25.362913 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260927 03:25:25.363653 14447 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35283
I20260927 03:25:25.363756 14447 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:25.363936 14447 heartbeater.cc:511] Master 127.0.98.62:35283 requested a full tablet report, sending...
I20260927 03:25:25.364360 14122 ts_manager.cc:194] Registered new tserver with Master: 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143)
I20260927 03:25:25.364745 14122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45465
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:25:25.385267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35283
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:25.465703 14452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:25.465924 14452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:25.465945 14452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:25.467080 14452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:25.467116 14452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:25.467141 14452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:25:25.468529 14452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:25.468709 14452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:25.468892 14452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:25.474422 14460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:25.474505 14458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:25.474453 14457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:25.474913 14452 server_base.cc:1034] running on GCE node
I20260927 03:25:25.475123 14452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:25.475374 14452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:25.476534 14452 hybrid_clock.cc:648] HybridClock initialized: now 1790479525476520 us; error 29 us; skew 500 ppm
Sep 27 03:25:25 dist-test-slave-gw2v krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:25.479063 14452 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:25:25.479547 14452 webserver.cc:460] Webserver started at http://127.0.98.3:36319/ using document root <none> and password file <none>
I20260927 03:25:25.479758 14452 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:25.479805 14452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:25.479929 14452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:25:25.480999 14452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "76b1f88e279b4729b54df390f28857b0"
format_stamp: "Formatted at 2026-09-27 03:25:25 on dist-test-slave-gw2v"
server_key: "f1a472b8b065b0ad68245f2e9c322349"
server_key_iv: "7cf319b8ffb10b8deb1da48d84ba85da"
server_key_version: "encryptionkey@0"
I20260927 03:25:25.481329 14452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "76b1f88e279b4729b54df390f28857b0"
format_stamp: "Formatted at 2026-09-27 03:25:25 on dist-test-slave-gw2v"
server_key: "f1a472b8b065b0ad68245f2e9c322349"
server_key_iv: "7cf319b8ffb10b8deb1da48d84ba85da"
server_key_version: "encryptionkey@0"
I20260927 03:25:25.482581 14452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:25:25.483381 14467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:25.483560 14452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:25:25.483652 14452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "76b1f88e279b4729b54df390f28857b0"
format_stamp: "Formatted at 2026-09-27 03:25:25 on dist-test-slave-gw2v"
server_key: "f1a472b8b065b0ad68245f2e9c322349"
server_key_iv: "7cf319b8ffb10b8deb1da48d84ba85da"
server_key_version: "encryptionkey@0"
I20260927 03:25:25.483718 14452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:25.496394 14452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:25.498476 14452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:25.498607 14452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:25.498842 14452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:25.499164 14452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:25.499205 14452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:25.499234 14452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:25.499259 14452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:25.504925 14452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34497
I20260927 03:25:25.504990 14580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34497 every 8 connection(s)
I20260927 03:25:25.505290 14452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:25:25.510088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 14452
I20260927 03:25:25.510208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:25:25.510689 392 external_mini_cluster.cc:1428] Setting key db8e58929a4f9a87420e7504b6180963
Sep 27 03:25:25 dist-test-slave-gw2v krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:25.514643 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.506006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50171 (local address 127.0.98.62:35283)
0927 03:25:25.506063 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:25.506065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:25.506248 (+ 183us) server_negotiation.cc:407] Connection header received
0927 03:25:25.506715 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:25.506716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:25.506728 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:25.506807 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:25.507750 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.508290 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.509480 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.509774 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.509842 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.512256 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:25.512257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:25.512257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:25.512265 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:25.513378 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:25.513741 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:25.513743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:25.513743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:25.513801 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:25.514028 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:25.514029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:25.514029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:25.514143 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:25.514199 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:25.514467 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 03:25:25.514557 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:25:25.514848 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.505960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:50171)
0927 03:25:25.506123 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:25.506166 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:25.506635 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:25.506896 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:25.506899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:25.507455 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:25.507649 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.507654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.508397 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.508398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.509395 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.509398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.509853 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.509854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.510059 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.510578 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:25.510589 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:25.512171 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:25.513460 (+ 1289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:25.513466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:25.513469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:25.513677 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:25.513864 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:25.513864 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:25.513865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:25.513974 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:25.514195 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:25.514201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:25.514339 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 03:25:25.514437 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 03:25:25.514606 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260927 03:25:25.515283 14581 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35283
I20260927 03:25:25.515383 14581 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:25.515570 14581 heartbeater.cc:511] Master 127.0.98.62:35283 requested a full tablet report, sending...
I20260927 03:25:25.515887 14122 ts_manager.cc:194] Registered new tserver with Master: 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497)
I20260927 03:25:25.516247 14122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50171
I20260927 03:25:25.522708 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:25:25.530251 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.524204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41840 (local address 127.0.98.62:35283)
0927 03:25:25.524242 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:25.524244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:25.524371 (+ 127us) server_negotiation.cc:407] Connection header received
0927 03:25:25.524429 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:25.524430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:25.524440 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:25.524487 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:25.525104 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.525616 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.526325 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.526515 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.526569 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.527895 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:25.527896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:25.527897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:25.527904 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:25.529181 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:25.529499 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:25.529500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:25.529501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:25.529559 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:25.529717 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:25.529717 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:25.529718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:25.529866 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:25.529964 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:25.530060 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 03:25:25.530141 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:25:25.531880 14122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:25:25.533434 14122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:25:25.539975 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.535700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34497 (local address 127.0.0.1:43474)
0927 03:25:25.536030 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:25.536075 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:25.536151 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:25.536416 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:25.536417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:25.536419 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:25.536480 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.536482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.537849 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.537850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.538995 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.538997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.539660 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.539661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.539743 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.539758 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:25:25.539828 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:25:25.539888 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":3,"thread_start_us":62,"threads_started":1}
I20260927 03:25:25.539999 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.535869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.0.98.3:34497)
0927 03:25:25.536290 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:25.536294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:25.536302 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:25:25.536332 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:25.536334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:25.536345 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:25.536407 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:25.536628 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.537750 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.539080 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.539582 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.539646 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.539712 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:25.539849 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 03:25:25.539889 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":66,"threads_started":1}
I20260927 03:25:25.540300 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.536050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51358 (local address 127.0.98.2:46143)
0927 03:25:25.536245 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:25.536250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:25.536259 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:25:25.536298 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:25.536300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:25.536312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:25.536376 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:25.536560 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.537477 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.539066 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.539808 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.539878 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.539933 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:25.540183 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 03:25:25.540228 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":81,"threads_started":1}
I20260927 03:25:25.540220 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.535700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46281 (local address 127.0.0.1:40174)
0927 03:25:25.536030 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:25.536075 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:25.536151 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:25.536744 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:25.536745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:25.536747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:25.536788 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.536790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.538017 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.538018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.538992 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.538995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.539972 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.539972 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.540035 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.540040 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:25:25.540089 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:25:25.540136 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":22,"thread_start_us":69,"threads_started":1}
I20260927 03:25:25.540550 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.535920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46143 (local address 127.0.0.1:51358)
0927 03:25:25.536093 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:25.536104 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:25.536155 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:25.536388 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:25.536392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:25.536394 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:25.536475 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.536477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.537567 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.537568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.538978 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.538981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.539970 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.539971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.540046 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.540049 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:25:25.540089 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 03:25:25.540136 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":17,"thread_start_us":49,"threads_started":1}
I20260927 03:25:25.540970 14515 tablet_service.cc:1467] Processing CreateTablet for tablet f0b3dda671ec427ab7d99f77b14fd9f2 (DEFAULT_TABLE table=test-table [id=3d8c4756271d4fee894215647d7d5006]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:25:25.541153 14381 tablet_service.cc:1467] Processing CreateTablet for tablet f0b3dda671ec427ab7d99f77b14fd9f2 (DEFAULT_TABLE table=test-table [id=3d8c4756271d4fee894215647d7d5006]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:25:25.541389 14515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b3dda671ec427ab7d99f77b14fd9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:25.541424 14381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b3dda671ec427ab7d99f77b14fd9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:25.543052 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.535810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40174 (local address 127.0.98.1:46281)
0927 03:25:25.536530 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:25.536534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:25.536543 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:25:25.536578 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:25.536580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:25.536595 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:25.536663 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:25.536856 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.537927 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.539082 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.539808 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.539878 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.539933 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:25.542927 (+ 2994us) server_negotiation.cc:299] Negotiation successful
0927 03:25:25.542982 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":98,"threads_started":1}
I20260927 03:25:25.544782 14596 tablet_bootstrap.cc:492] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0: Bootstrap starting.
I20260927 03:25:25.545318 14247 tablet_service.cc:1467] Processing CreateTablet for tablet f0b3dda671ec427ab7d99f77b14fd9f2 (DEFAULT_TABLE table=test-table [id=3d8c4756271d4fee894215647d7d5006]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:25:25.545413 14596 tablet_bootstrap.cc:654] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:25.545599 14247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b3dda671ec427ab7d99f77b14fd9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:25.545850 14596 log.cc:824] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:25.546212 14595 tablet_bootstrap.cc:492] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0: Bootstrap starting.
I20260927 03:25:25.546655 14596 tablet_bootstrap.cc:492] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0: No bootstrap required, opened a new log
I20260927 03:25:25.546731 14596 ts_tablet_manager.cc:1397] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:25:25.546733 14595 tablet_bootstrap.cc:654] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:25.547309 14595 log.cc:824] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:25.547963 14595 tablet_bootstrap.cc:492] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0: No bootstrap required, opened a new log
I20260927 03:25:25.548030 14595 ts_tablet_manager.cc:1397] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:25:25.548755 14596 raft_consensus.cc:348] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } } peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } }
I20260927 03:25:25.548919 14599 tablet_bootstrap.cc:492] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: Bootstrap starting.
I20260927 03:25:25.549046 14596 raft_consensus.cc:374] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:25.549113 14596 raft_consensus.cc:729] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b1f88e279b4729b54df390f28857b0, State: Initialized, Role: FOLLOWER
I20260927 03:25:25.549268 14596 consensus_queue.cc:260] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } } peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } }
I20260927 03:25:25.549537 14596 ts_tablet_manager.cc:1428] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:25:25.549606 14599 tablet_bootstrap.cc:654] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:25.549652 14581 heartbeater.cc:503] Master 127.0.98.62:35283 was elected leader, sending a full tablet report...
I20260927 03:25:25.549853 14595 raft_consensus.cc:348] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } } peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } }
I20260927 03:25:25.550058 14595 raft_consensus.cc:374] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:25.550136 14595 raft_consensus.cc:729] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba88939b2cb4e60b90af9e2275f57b0, State: Initialized, Role: FOLLOWER
I20260927 03:25:25.550138 14599 log.cc:824] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:25.550266 14595 consensus_queue.cc:260] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } } peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } }
I20260927 03:25:25.550535 14595 ts_tablet_manager.cc:1428] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:25:25.550573 14447 heartbeater.cc:503] Master 127.0.98.62:35283 was elected leader, sending a full tablet report...
I20260927 03:25:25.551044 14599 tablet_bootstrap.cc:492] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: No bootstrap required, opened a new log
I20260927 03:25:25.551139 14599 ts_tablet_manager.cc:1397] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:25:25.552615 14599 raft_consensus.cc:348] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } } peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } }
I20260927 03:25:25.552752 14599 raft_consensus.cc:374] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:25.552772 14599 raft_consensus.cc:729] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d72e99c6079842e6a6ba2f37bdc136f9, State: Initialized, Role: FOLLOWER
I20260927 03:25:25.552846 14599 consensus_queue.cc:260] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } } peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } }
I20260927 03:25:25.553063 14599 ts_tablet_manager.cc:1428] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:25:25.553114 14313 heartbeater.cc:503] Master 127.0.98.62:35283 was elected leader, sending a full tablet report...
W20260927 03:25:25.603639 14448 tablet.cc:2369] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:25:25.704473 14314 tablet.cc:2369] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:25:25.755903 14582 tablet.cc:2369] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:25:25.842412 14602 raft_consensus.cc:482] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:25:25.842550 14602 raft_consensus.cc:504] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } } peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } }
I20260927 03:25:25.842978 14602 leader_election.cc:288] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d72e99c6079842e6a6ba2f37bdc136f9 (127.0.98.1:46281), 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497)
I20260927 03:25:25.845795 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.843232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49903 (local address 127.0.98.1:46281)
0927 03:25:25.843282 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:25.843284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:25.843412 (+ 128us) server_negotiation.cc:407] Connection header received
0927 03:25:25.843444 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:25.843445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:25.843457 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:25.843510 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:25.843679 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.844123 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.845117 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.845493 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.845545 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.845555 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:25.845691 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 03:25:25.845727 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:25:25.845799 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.843186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46281 (local address 127.0.98.2:49903)
0927 03:25:25.843325 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:25.843333 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:25.843382 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:25.843533 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:25.843534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:25.843537 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:25.843611 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.843613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.844192 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.844193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.845031 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.845034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.845565 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.845566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.845631 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.845635 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:25:25.845675 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 03:25:25.845708 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":42,"thread_start_us":77,"threads_started":1}
I20260927 03:25:25.845832 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.843318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48859 (local address 127.0.98.3:34497)
0927 03:25:25.843365 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:25.843367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:25.843376 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:25:25.843402 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:25.843402 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:25.843410 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:25.843479 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:25.843646 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.844132 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.845160 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.845518 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.845584 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.845594 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:25.845734 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 03:25:25.845774 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:25:25.845880 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.843186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34497 (local address 127.0.98.2:48859)
0927 03:25:25.843241 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:25.843253 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:25.843299 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:25.843490 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:25.843491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:25.843493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:25.843581 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.843583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.844210 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.844211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.845067 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:25.845070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.845608 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.845609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:25.845669 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.845672 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:25:25.845723 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:25:25.845760 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":93}
I20260927 03:25:25.846218 14267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b3dda671ec427ab7d99f77b14fd9f2" candidate_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" is_pre_election: true
I20260927 03:25:25.846217 14535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b3dda671ec427ab7d99f77b14fd9f2" candidate_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b1f88e279b4729b54df390f28857b0" is_pre_election: true
I20260927 03:25:25.846379 14267 raft_consensus.cc:2454] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ba88939b2cb4e60b90af9e2275f57b0 in term 0.
I20260927 03:25:25.846379 14535 raft_consensus.cc:2454] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ba88939b2cb4e60b90af9e2275f57b0 in term 0.
I20260927 03:25:25.846599 14336 leader_election.cc:302] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba88939b2cb4e60b90af9e2275f57b0, 76b1f88e279b4729b54df390f28857b0; no voters:
I20260927 03:25:25.846776 14602 raft_consensus.cc:2781] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:25:25.846858 14602 raft_consensus.cc:482] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:25:25.846884 14602 raft_consensus.cc:3037] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:25.847680 14602 raft_consensus.cc:504] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } } peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } }
I20260927 03:25:25.847798 14602 leader_election.cc:288] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [CANDIDATE]: Term 1 election: Requested vote from peers d72e99c6079842e6a6ba2f37bdc136f9 (127.0.98.1:46281), 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497)
I20260927 03:25:25.847957 14267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b3dda671ec427ab7d99f77b14fd9f2" candidate_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72e99c6079842e6a6ba2f37bdc136f9"
I20260927 03:25:25.847955 14535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b3dda671ec427ab7d99f77b14fd9f2" candidate_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b1f88e279b4729b54df390f28857b0"
I20260927 03:25:25.848033 14535 raft_consensus.cc:3037] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:25.848037 14267 raft_consensus.cc:3037] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:25.848770 14267 raft_consensus.cc:2454] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ba88939b2cb4e60b90af9e2275f57b0 in term 1.
I20260927 03:25:25.848773 14535 raft_consensus.cc:2454] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ba88939b2cb4e60b90af9e2275f57b0 in term 1.
I20260927 03:25:25.848994 14336 leader_election.cc:302] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba88939b2cb4e60b90af9e2275f57b0, 76b1f88e279b4729b54df390f28857b0; no voters:
I20260927 03:25:25.849118 14602 raft_consensus.cc:2781] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:25:25.849282 14602 raft_consensus.cc:686] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 1 LEADER]: Becoming Leader. State: Replica: 2ba88939b2cb4e60b90af9e2275f57b0, State: Running, Role: LEADER
I20260927 03:25:25.849368 14602 consensus_queue.cc:237] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } } peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } }
I20260927 03:25:25.850147 14121 catalog_manager.cc:5495] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } health_report { overall_health: UNKNOWN } } }
I20260927 03:25:25.893584 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.891168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51362 (local address 127.0.98.2:46143)
0927 03:25:25.891224 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:25.891225 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:25.891233 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:25:25.891306 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:25.891307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:25.891327 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:25.891388 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:25:25.891503 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.892015 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.892796 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:25.893040 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:25.893097 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:25.893236 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:25:25.893346 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:25:25.893402 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:25.893485 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 03:25:25.893503 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:25:25.895465 14535 raft_consensus.cc:1261] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0 [term 1 FOLLOWER]: Refusing update from remote peer 2ba88939b2cb4e60b90af9e2275f57b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:25:25.895489 14267 raft_consensus.cc:1261] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Refusing update from remote peer 2ba88939b2cb4e60b90af9e2275f57b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:25:25.895824 14605 consensus_queue.cc:1035] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:25:25.895964 14602 consensus_queue.cc:1035] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:25:25.898123 14608 mvcc.cc:205] Tried to move back new op lower bound from 7333804138064973824 to 7333804137879588864. Current Snapshot: MvccSnapshot[applied={T|T < 7333804138064973824}]
I20260927 03:25:25.898736 14610 mvcc.cc:205] Tried to move back new op lower bound from 7333804138064973824 to 7333804137879588864. Current Snapshot: MvccSnapshot[applied={T|T < 7333804138064973824}]
I20260927 03:25:25.900519 14121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:25:25.900614 14121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:25:25.901378 14121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: Sending DeleteTablet for 3 replicas of tablet f0b3dda671ec427ab7d99f77b14fd9f2
I20260927 03:25:25.901672 14247 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b3dda671ec427ab7d99f77b14fd9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:25:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40174
I20260927 03:25:25.901711 14381 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b3dda671ec427ab7d99f77b14fd9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:25:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51358
I20260927 03:25:25.901816 14515 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b3dda671ec427ab7d99f77b14fd9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:25:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43474
I20260927 03:25:25.902015 14620 tablet_replica.cc:330] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0: stopping tablet replica
I20260927 03:25:25.902034 14619 tablet_replica.cc:330] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0: stopping tablet replica
I20260927 03:25:25.902091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 14184
I20260927 03:25:25.902128 14619 raft_consensus.cc:2229] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:25:25.902765 14619 raft_consensus.cc:2258] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:25:25.903235 14619 ts_tablet_manager.cc:1905] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:25:25.905292 14619 ts_tablet_manager.cc:1918] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:25:25.905407 14619 log.cc:1196] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/f0b3dda671ec427ab7d99f77b14fd9f2
I20260927 03:25:25.905685 14619 ts_tablet_manager.cc:1939] T f0b3dda671ec427ab7d99f77b14fd9f2 P 2ba88939b2cb4e60b90af9e2275f57b0: Deleting consensus metadata
I20260927 03:25:25.906248 14109 catalog_manager.cc:4841] TS 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143): tablet f0b3dda671ec427ab7d99f77b14fd9f2 (table test-table [id=3d8c4756271d4fee894215647d7d5006]) successfully deleted
I20260927 03:25:25.906905 14620 raft_consensus.cc:2229] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:25:25.907073 14620 raft_consensus.cc:2258] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:25:25.907512 14620 ts_tablet_manager.cc:1905] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 03:25:25.908236 14109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46281 (error 108)
I20260927 03:25:25.908344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 14318
I20260927 03:25:25.908985 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:25.908734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46281 (local address 127.0.0.1:40184)
0927 03:25:25.908894 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:25.908949 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260927 03:25:25.909034 14620 ts_tablet_manager.cc:1918] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:25:25.909089 14620 log.cc:1196] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/f0b3dda671ec427ab7d99f77b14fd9f2
I20260927 03:25:25.909332 14620 ts_tablet_manager.cc:1939] T f0b3dda671ec427ab7d99f77b14fd9f2 P 76b1f88e279b4729b54df390f28857b0: Deleting consensus metadata
W20260927 03:25:25.909396 14109 catalog_manager.cc:4584] TS d72e99c6079842e6a6ba2f37bdc136f9 (127.0.98.1:46281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0b3dda671ec427ab7d99f77b14fd9f2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46281: connect: Connection refused (error 111)
I20260927 03:25:25.909823 14108 catalog_manager.cc:4841] TS 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497): tablet f0b3dda671ec427ab7d99f77b14fd9f2 (table test-table [id=3d8c4756271d4fee894215647d7d5006]) successfully deleted
I20260927 03:25:25.913371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 14452
I20260927 03:25:25.917640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 14090
I20260927 03:25:25.922310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35283
--webserver_interface=127.0.98.62
--webserver_port=36155
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:26.003662 14622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:26.003872 14622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:26.003901 14622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:26.005069 14622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:25:26.005113 14622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:26.005126 14622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:26.005137 14622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:25:26.005148 14622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:25:26.006609 14622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:26.006855 14622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:26.007085 14622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:26.015653 14628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:26.015646 14629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:26.015643 14631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:26.015707 14622 server_base.cc:1034] running on GCE node
I20260927 03:25:26.016150 14622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:26.016536 14622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:26.017742 14622 hybrid_clock.cc:648] HybridClock initialized: now 1790479526017727 us; error 31 us; skew 500 ppm
Sep 27 03:25:26 dist-test-slave-gw2v krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:26.024216 14622 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:25:26.024691 14622 webserver.cc:460] Webserver started at http://127.0.98.62:36155/ using document root <none> and password file <none>
I20260927 03:25:26.024928 14622 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:26.024993 14622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:26.026355 14622 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:25:26.027168 14638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:26.027328 14622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:25:26.027407 14622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "4d407a2dc4614e3abadc780570705a9e"
format_stamp: "Formatted at 2026-09-27 03:25:24 on dist-test-slave-gw2v"
server_key: "990a8319f06426d659cbc8fc9716f82d"
server_key_iv: "e2f0e5d6bbf8b93f76cf5a1db62fd013"
server_key_version: "encryptionkey@0"
I20260927 03:25:26.027817 14622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:26.049687 14622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:26.051991 14622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:26.052141 14622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:26.056123 14622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35283
I20260927 03:25:26.056211 14690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35283 every 8 connection(s)
I20260927 03:25:26.056571 14622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:25:26.057510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 14622
I20260927 03:25:26.057771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46281
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35283
--webserver_port=36891
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:25:26.058342 14691 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:25:26.059177 14691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: Bootstrap starting.
I20260927 03:25:26.068823 14691 log.cc:824] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:26.071616 14691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:25:26.071949 14691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: Bootstrap complete.
I20260927 03:25:26.075162 14691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260927 03:25:26.075503 14691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d407a2dc4614e3abadc780570705a9e, State: Initialized, Role: FOLLOWER
I20260927 03:25:26.075668 14691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260927 03:25:26.075732 14691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:25:26.075780 14691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:25:26.075826 14691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:25:26.077216 14691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260927 03:25:26.077335 14691 leader_election.cc:302] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d407a2dc4614e3abadc780570705a9e; no voters:
I20260927 03:25:26.077507 14691 leader_election.cc:288] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:25:26.077584 14695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:25:26.077764 14691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:25:26.077809 14695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 2 LEADER]: Becoming Leader. State: Replica: 4d407a2dc4614e3abadc780570705a9e, State: Running, Role: LEADER
I20260927 03:25:26.077981 14695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260927 03:25:26.078286 14696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4d407a2dc4614e3abadc780570705a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } } }
I20260927 03:25:26.078358 14696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:26.078751 14697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d407a2dc4614e3abadc780570705a9e. Latest consensus state: current_term: 2 leader_uuid: "4d407a2dc4614e3abadc780570705a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } } }
I20260927 03:25:26.078888 14697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:26.079023 14703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:25:26.079892 14703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:25:26.079993 14703 catalog_manager.cc:1259] Loaded cluster ID: ac2d5f874b464eae85c6c6300fe44183
I20260927 03:25:26.080057 14703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:25:26.080303 14703 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 03:25:26.084928 14703 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 03:25:26.088054 14703 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:25:26.088295 14703 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: Loaded TSK: 0
I20260927 03:25:26.088528 14703 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:25:26.146618 14693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:26.146850 14693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:26.146886 14693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:26.148093 14693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:26.148130 14693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:26.148156 14693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:25:26.149660 14693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36891
--tserver_master_addrs=127.0.98.62:35283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:26.149868 14693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:26.150081 14693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:26.155215 14718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:26.155215 14719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:26.155326 14721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:26.155678 14693 server_base.cc:1034] running on GCE node
I20260927 03:25:26.155898 14693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:26.156147 14693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:26.157303 14693 hybrid_clock.cc:648] HybridClock initialized: now 1790479526157288 us; error 31 us; skew 500 ppm
Sep 27 03:25:26 dist-test-slave-gw2v krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:26.159953 14693 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:25:26.160442 14693 webserver.cc:460] Webserver started at http://127.0.98.1:36891/ using document root <none> and password file <none>
I20260927 03:25:26.160681 14693 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:26.160746 14693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:26.162235 14693 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:25:26.163003 14728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:26.163168 14693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:25:26.163245 14693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "d72e99c6079842e6a6ba2f37bdc136f9"
format_stamp: "Formatted at 2026-09-27 03:25:25 on dist-test-slave-gw2v"
server_key: "f748b207047a4e5a8dc2be9c136fdfbf"
server_key_iv: "e48c9b07ca9fe6f3cb95b174e92056b7"
server_key_version: "encryptionkey@0"
I20260927 03:25:26.163640 14693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:26.188413 14693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:26.190443 14693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:26.190582 14693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:26.190850 14693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:26.191365 14735 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:25:26.192785 14693 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:25:26.192845 14693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:25:26.192871 14693 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:25:26.193508 14693 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:25:26.193550 14693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:25:26.193619 14735 tablet_bootstrap.cc:492] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: Bootstrap starting.
I20260927 03:25:26.200690 14693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46281
I20260927 03:25:26.200766 14842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46281 every 8 connection(s)
I20260927 03:25:26.201370 14693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:25:26.204186 14735 log.cc:824] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:26.206610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 14693
I20260927 03:25:26.206693 14735 tablet_bootstrap.cc:492] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:25:26.206928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46143
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35283
--webserver_port=36633
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:25:26.206990 14735 tablet_bootstrap.cc:492] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: Bootstrap complete.
I20260927 03:25:26.207980 14735 ts_tablet_manager.cc:1397] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20260927 03:25:26.209151 14735 raft_consensus.cc:348] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } } peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } }
I20260927 03:25:26.209379 14735 raft_consensus.cc:729] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d72e99c6079842e6a6ba2f37bdc136f9, State: Initialized, Role: FOLLOWER
I20260927 03:25:26.209553 14735 consensus_queue.cc:260] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } } peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } }
I20260927 03:25:26.209863 14735 ts_tablet_manager.cc:1428] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
Sep 27 03:25:26 dist-test-slave-gw2v krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:26.213791 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.202037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:33843)
0927 03:25:26.202932 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:26.202982 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:26.203638 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:26.204161 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:26.204165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:26.204512 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:26.204788 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.204795 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.206395 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.206396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.207500 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.207502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.208128 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.208129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.208263 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.208818 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:26.208828 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:26.210451 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:26.212326 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:26.212332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:26.212335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:26.212549 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:26.212761 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:26.212762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:26.212762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:26.212867 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:26.213203 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:26.213209 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:26.213338 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 03:25:26.213428 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 03:25:26.213598 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":78,"threads_started":1}
I20260927 03:25:26.213791 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.202167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33843 (local address 127.0.98.62:35283)
0927 03:25:26.202370 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:26.202381 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:26.203110 (+ 729us) server_negotiation.cc:407] Connection header received
0927 03:25:26.203875 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:26.203883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:26.203946 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:26.204103 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:26.204909 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.206255 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.207584 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.208001 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.208103 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.210548 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:26.210552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:26.210552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:26.210564 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:26.212257 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:26.212621 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:26.212624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:26.212624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:26.212705 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:26.212922 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:26.212923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:26.212923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:26.213132 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:26.213201 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:26.213488 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 03:25:26.213600 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260927 03:25:26.214367 14843 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35283
I20260927 03:25:26.214458 14843 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:26.214630 14843 heartbeater.cc:511] Master 127.0.98.62:35283 requested a full tablet report, sending...
I20260927 03:25:26.215320 14655 ts_manager.cc:194] Registered new tserver with Master: d72e99c6079842e6a6ba2f37bdc136f9 (127.0.98.1:46281)
I20260927 03:25:26.216379 14655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33843
I20260927 03:25:26.217171 14843 heartbeater.cc:503] Master 127.0.98.62:35283 was elected leader, sending a full tablet report...
Sep 27 03:25:26 dist-test-slave-gw2v krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:25:26.224921 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.215894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46281 (local address 127.0.0.1:40192)
0927 03:25:26.216060 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:26.216105 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:26.216175 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:26.216497 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:26.216502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:26.216829 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:26.216891 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.216893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.218291 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.218292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.219017 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.219020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.219593 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.219596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.219642 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.220082 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:26.220086 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:26.221615 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:26.223735 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:26.223742 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:26.223744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:26.224002 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:26.224199 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:26.224200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:26.224200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:26.224285 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:26.224572 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:26.224578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:26.224626 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 03:25:26.224692 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:25:26.224796 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260927 03:25:26.224923 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.216115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.0.98.1:46281)
0927 03:25:26.216327 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:26.216332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:26.216341 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:25:26.216372 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:26.216374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:26.216389 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:26.216972 (+ 583us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:26.217118 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.218202 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.219083 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.219534 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.219599 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.221697 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:26.221706 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:26.221707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:26.221714 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:26.223655 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:26.224076 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:26.224080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:26.224080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:26.224144 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:26.224342 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:26.224343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:26.224344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:26.224508 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:26.224585 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:26.224738 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 03:25:26.224820 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":68,"threads_started":1}
I20260927 03:25:26.225338 14777 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b3dda671ec427ab7d99f77b14fd9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:25:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40192
I20260927 03:25:26.225858 14852 tablet_replica.cc:330] stopping tablet replica
I20260927 03:25:26.225953 14852 raft_consensus.cc:2229] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:25:26.226023 14852 raft_consensus.cc:2258] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:25:26.226303 14852 ts_tablet_manager.cc:1905] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:25:26.228339 14852 ts_tablet_manager.cc:1918] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:25:26.228425 14852 log.cc:1196] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal/wals/f0b3dda671ec427ab7d99f77b14fd9f2
I20260927 03:25:26.228653 14852 ts_tablet_manager.cc:1939] T f0b3dda671ec427ab7d99f77b14fd9f2 P d72e99c6079842e6a6ba2f37bdc136f9: Deleting consensus metadata
I20260927 03:25:26.229171 14642 catalog_manager.cc:4841] TS d72e99c6079842e6a6ba2f37bdc136f9 (127.0.98.1:46281): tablet f0b3dda671ec427ab7d99f77b14fd9f2 (table test-table [id=3d8c4756271d4fee894215647d7d5006]) successfully deleted
W20260927 03:25:26.291332 14848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:26.291548 14848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:26.291579 14848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:26.292737 14848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:26.292775 14848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:26.292817 14848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:25:26.294226 14848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36633
--tserver_master_addrs=127.0.98.62:35283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:26.294452 14848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:26.294667 14848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:26.299446 14858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:26.299467 14857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:26.299572 14860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:26.299530 14848 server_base.cc:1034] running on GCE node
I20260927 03:25:26.299827 14848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:26.300082 14848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:26.301245 14848 hybrid_clock.cc:648] HybridClock initialized: now 1790479526301230 us; error 27 us; skew 500 ppm
Sep 27 03:25:26 dist-test-slave-gw2v krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:26.304028 14848 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:25:26.304492 14848 webserver.cc:460] Webserver started at http://127.0.98.2:36633/ using document root <none> and password file <none>
I20260927 03:25:26.304693 14848 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:26.304759 14848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:26.306149 14848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:25:26.306778 14867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:26.307000 14848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:25:26.307086 14848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "2ba88939b2cb4e60b90af9e2275f57b0"
format_stamp: "Formatted at 2026-09-27 03:25:25 on dist-test-slave-gw2v"
server_key: "53b71e254aed891b4cbc08b293190836"
server_key_iv: "2af91cc0120951223f351425fdfa9776"
server_key_version: "encryptionkey@0"
I20260927 03:25:26.307427 14848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:26.321877 14848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:26.324241 14848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:26.324385 14848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:26.324615 14848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:26.324918 14848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:26.324959 14848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:26.324980 14848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:26.324991 14848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:26.330276 14848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46143
I20260927 03:25:26.330336 14980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46143 every 8 connection(s)
I20260927 03:25:26.330641 14848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:25:26.333173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 14848
I20260927 03:25:26.333427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34497
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35283
--webserver_port=36319
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:25:26 dist-test-slave-gw2v krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:26.340842 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.331349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39229 (local address 127.0.98.62:35283)
0927 03:25:26.331422 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:26.331423 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:26.331618 (+ 195us) server_negotiation.cc:407] Connection header received
0927 03:25:26.332126 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:26.332127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:26.332140 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:26.332197 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:26.333200 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.333891 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.335184 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.335405 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.335464 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.337947 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:26.337949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:26.337949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:26.337957 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:26.339396 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:26.339874 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:26.339874 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:26.339875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:26.339964 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:26.340229 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:26.340229 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:26.340229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:26.340329 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:26.340371 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:26.340701 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 03:25:26.340772 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:25:26.341122 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.331355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:39229)
0927 03:25:26.331500 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:26.331534 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:26.332015 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:26.332253 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:26.332256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:26.332812 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:26.333073 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.333081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.334000 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.334002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.335104 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.335107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.335477 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.335477 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.335594 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.336113 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:26.336123 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:26.337764 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:26.339477 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:26.339485 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:26.339488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:26.339803 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:26.340023 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:26.340024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:26.340024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:26.340169 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:26.340381 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:26.340386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:26.340568 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 03:25:26.340687 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 03:25:26.340867 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260927 03:25:26.341742 14981 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35283
I20260927 03:25:26.341866 14981 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:26.342082 14981 heartbeater.cc:511] Master 127.0.98.62:35283 requested a full tablet report, sending...
I20260927 03:25:26.342497 14655 ts_manager.cc:194] Registered new tserver with Master: 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143)
I20260927 03:25:26.342924 14655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39229
W20260927 03:25:26.416754 14984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:26.416998 14984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:26.417033 14984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:26.418273 14984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:26.418324 14984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:26.418363 14984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:25:26.419981 14984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36319
--tserver_master_addrs=127.0.98.62:35283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:26.420233 14984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:26.420454 14984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:26.425771 14990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:26.425773 14992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:26.426075 14984 server_base.cc:1034] running on GCE node
W20260927 03:25:26.425865 14989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:26.426386 14984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:26.426676 14984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:26.427832 14984 hybrid_clock.cc:648] HybridClock initialized: now 1790479526427867 us; error 92 us; skew 500 ppm
Sep 27 03:25:26 dist-test-slave-gw2v krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:26.430661 14984 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:25:26.431226 14984 webserver.cc:460] Webserver started at http://127.0.98.3:36319/ using document root <none> and password file <none>
I20260927 03:25:26.431409 14984 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:26.431491 14984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:26.432821 14984 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:25:26.433368 14999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:26.433555 14984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:25:26.433636 14984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "76b1f88e279b4729b54df390f28857b0"
format_stamp: "Formatted at 2026-09-27 03:25:25 on dist-test-slave-gw2v"
server_key: "f1a472b8b065b0ad68245f2e9c322349"
server_key_iv: "7cf319b8ffb10b8deb1da48d84ba85da"
server_key_version: "encryptionkey@0"
I20260927 03:25:26.434002 14984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:26.457270 14984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:26.459381 14984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:26.459524 14984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:26.459739 14984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:26.460040 14984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:26.460076 14984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:26.460105 14984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:26.460122 14984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:26.465541 14984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34497
I20260927 03:25:26.465593 15112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34497 every 8 connection(s)
I20260927 03:25:26.465919 14984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:25:26.469146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 14984
Sep 27 03:25:26 dist-test-slave-gw2v krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:26.475929 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.466616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55729 (local address 127.0.98.62:35283)
0927 03:25:26.466680 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:26.466682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:26.466933 (+ 251us) server_negotiation.cc:407] Connection header received
0927 03:25:26.467433 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:26.467434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:26.467446 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:26.467505 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:26.468409 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.468899 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.470145 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.470412 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.470483 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.472807 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:26.472809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:26.472809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:26.472816 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:26.474242 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:26.474753 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:26.474754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:26.474755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:26.474869 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:26.475192 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:26.475194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:26.475194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:26.475332 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:26.475373 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:26.475748 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 03:25:26.475835 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:25:26.476459 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.466609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:55729)
0927 03:25:26.466770 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:26.466846 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:26.467339 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:26.467577 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:26.467581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:26.468129 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:26.468316 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.468321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.468983 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.468984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.470033 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.470035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.470497 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.470498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.470596 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.471119 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:26.471129 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:26.472713 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:26.474331 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:26.474338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:26.474341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:26.474633 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:26.474966 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:26.474967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:26.474967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:26.475104 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:26.475419 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:26.475426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:26.475591 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 03:25:26.475703 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 03:25:26.475885 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260927 03:25:26.476625 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.469533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41852 (local address 127.0.98.62:35283)
0927 03:25:26.469628 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:26.469630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:26.469640 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:25:26.469695 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:26.469696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:26.469704 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:26.469754 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:26.470363 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.470981 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.471939 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.472140 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.472192 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.473349 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:26.473350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:26.473350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:26.473358 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:26.474999 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:26.475818 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:26.475819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:26.475820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:26.475876 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:26.476167 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:26.476168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:26.476168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:26.476267 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:26.476435 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:26.476470 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 03:25:26.476542 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":53,"threads_started":1}
I20260927 03:25:26.477211 15113 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35283
I20260927 03:25:26.477319 15113 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:26.477521 15113 heartbeater.cc:511] Master 127.0.98.62:35283 requested a full tablet report, sending...
I20260927 03:25:26.477872 14655 ts_manager.cc:194] Registered new tserver with Master: 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497)
I20260927 03:25:26.478248 14655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55729
I20260927 03:25:26.488499 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:25:26.491232 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.488923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.0.98.62:35283)
0927 03:25:26.488986 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:26.488988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:26.489066 (+ 78us) server_negotiation.cc:407] Connection header received
0927 03:25:26.489128 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:26.489129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:26.489164 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:26.489224 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:25:26.489341 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.489847 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.490554 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.490746 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.490792 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.490971 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:25:26.491070 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:25:26.491131 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:26.491147 (+ 16us) server_negotiation.cc:299] Negotiation successful
0927 03:25:26.491166 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:25:26.491532 14655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:25:26.493178 14655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:25:26.495414 14777 tablet_service.cc:1467] Processing CreateTablet for tablet 1ee79cdca4a84ecc87f064ddb5719f87 (DEFAULT_TABLE table=test-table [id=2d2baf1ad169476c8c9678023e9b0333]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:25:26.495613 14777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ee79cdca4a84ecc87f064ddb5719f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:26.497141 14735 tablet_bootstrap.cc:492] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9: Bootstrap starting.
I20260927 03:25:26.497445 14735 tablet_bootstrap.cc:654] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:26.498243 14735 tablet_bootstrap.cc:492] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9: No bootstrap required, opened a new log
I20260927 03:25:26.498306 14735 ts_tablet_manager.cc:1397] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:25:26.498440 14735 raft_consensus.cc:348] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:26.498509 14735 raft_consensus.cc:374] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:26.498554 14735 raft_consensus.cc:729] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d72e99c6079842e6a6ba2f37bdc136f9, State: Initialized, Role: FOLLOWER
I20260927 03:25:26.498594 14735 consensus_queue.cc:260] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:26.498674 14735 ts_tablet_manager.cc:1428] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:26.499245 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.495093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46143 (local address 127.0.0.1:51376)
0927 03:25:26.495141 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:26.495152 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:26.495189 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:26.495613 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:26.495614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:26.495616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:26.495652 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.495653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.496864 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.496864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.497901 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.497904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.498979 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.498980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.499053 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.499075 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 03:25:26.499129 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 03:25:26.499176 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 03:25:26.499444 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.495193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.0.98.2:46143)
0927 03:25:26.495401 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:26.495406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:26.495416 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:25:26.495456 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:26.495458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:26.495472 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:26.495552 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:26.495724 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.496767 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.498040 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.498871 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.498938 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.499009 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:26.499323 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 03:25:26.499366 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":96,"threads_started":1}
I20260927 03:25:26.500109 14915 tablet_service.cc:1467] Processing CreateTablet for tablet 1ee79cdca4a84ecc87f064ddb5719f87 (DEFAULT_TABLE table=test-table [id=2d2baf1ad169476c8c9678023e9b0333]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:25:26.500378 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.495136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34497 (local address 127.0.0.1:43490)
0927 03:25:26.497175 (+ 2039us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:26.497186 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:26.497234 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:26.497425 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:26.497426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:26.497428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:26.497467 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.497468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.498391 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.498391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.499396 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.499399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.500168 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.500169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.500229 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.500233 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:25:26.500279 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:25:26.500318 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2021,"thread_start_us":95,"threads_started":1}
I20260927 03:25:26.500398 14915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ee79cdca4a84ecc87f064ddb5719f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:26.500597 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.495325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.0.98.3:34497)
0927 03:25:26.495472 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:26.495476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:26.497291 (+ 1815us) server_negotiation.cc:407] Connection header received
0927 03:25:26.497335 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:26.497336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:26.497349 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:26.497413 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:26.497518 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.498280 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.499488 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.500090 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.500153 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.500207 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:26.500474 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 03:25:26.500517 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260927 03:25:26.501168 15047 tablet_service.cc:1467] Processing CreateTablet for tablet 1ee79cdca4a84ecc87f064ddb5719f87 (DEFAULT_TABLE table=test-table [id=2d2baf1ad169476c8c9678023e9b0333]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:25:26.501487 15047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ee79cdca4a84ecc87f064ddb5719f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:26.503252 15123 tablet_bootstrap.cc:492] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0: Bootstrap starting.
I20260927 03:25:26.503827 15123 tablet_bootstrap.cc:654] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:26.504130 15124 tablet_bootstrap.cc:492] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0: Bootstrap starting.
I20260927 03:25:26.504549 15123 log.cc:824] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:26.504802 15124 tablet_bootstrap.cc:654] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:26.505225 15124 log.cc:824] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:26.505383 15123 tablet_bootstrap.cc:492] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0: No bootstrap required, opened a new log
I20260927 03:25:26.505446 15123 ts_tablet_manager.cc:1397] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:25:26.505932 15124 tablet_bootstrap.cc:492] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0: No bootstrap required, opened a new log
I20260927 03:25:26.506003 15124 ts_tablet_manager.cc:1397] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:25:26.507345 15123 raft_consensus.cc:348] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:26.507534 15123 raft_consensus.cc:374] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:26.507568 15123 raft_consensus.cc:729] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba88939b2cb4e60b90af9e2275f57b0, State: Initialized, Role: FOLLOWER
I20260927 03:25:26.507720 15123 consensus_queue.cc:260] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:26.507877 15124 raft_consensus.cc:348] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:26.508008 15123 ts_tablet_manager.cc:1428] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:25:26.508037 15124 raft_consensus.cc:374] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:26.508030 14981 heartbeater.cc:503] Master 127.0.98.62:35283 was elected leader, sending a full tablet report...
I20260927 03:25:26.508076 15124 raft_consensus.cc:729] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b1f88e279b4729b54df390f28857b0, State: Initialized, Role: FOLLOWER
I20260927 03:25:26.508176 15124 consensus_queue.cc:260] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:26.508431 15124 ts_tablet_manager.cc:1428] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 03:25:26.508464 15113 heartbeater.cc:503] Master 127.0.98.62:35283 was elected leader, sending a full tablet report...
I20260927 03:25:26.540835 15128 raft_consensus.cc:482] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:25:26.540975 15128 raft_consensus.cc:504] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:26.541401 15128 leader_election.cc:288] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143), d72e99c6079842e6a6ba2f37bdc136f9 (127.0.98.1:46281)
I20260927 03:25:26.544435 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.541636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56159 (local address 127.0.98.2:46143)
0927 03:25:26.541681 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:26.541683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:26.541693 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:25:26.541722 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:26.541723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:26.541734 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:26.541786 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:26.542039 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.542581 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.543712 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.544103 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.544157 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.544168 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:26.544324 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 03:25:26.544365 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:25:26.544435 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.541546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46143 (local address 127.0.98.3:56159)
0927 03:25:26.541592 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:26.541606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:26.541656 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:26.541828 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:26.541829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:26.541830 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:26.541951 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.541954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.542657 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.542658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.543619 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.543622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.544179 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.544180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.544262 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.544266 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:25:26.544314 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:25:26.544356 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 03:25:26.544512 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.541796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45755 (local address 127.0.98.1:46281)
0927 03:25:26.541842 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:26.541843 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:26.541965 (+ 122us) server_negotiation.cc:407] Connection header received
0927 03:25:26.541999 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:26.542000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:26.542027 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:26.542082 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:26.542234 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.542770 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.543745 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.544195 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.544270 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.544362 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:26.544409 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 03:25:26.544454 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:25:26.544574 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.541647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46281 (local address 127.0.98.3:45755)
0927 03:25:26.541885 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:26.541895 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:26.541952 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:26.542103 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:26.542104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:26.542106 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:26.542174 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.542176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.542873 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.542874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.543642 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:26.543644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.544276 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.544277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:26.544351 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.544355 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:25:26.544404 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:25:26.544443 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":132,"threads_started":1}
I20260927 03:25:26.544818 14935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee79cdca4a84ecc87f064ddb5719f87" candidate_uuid: "76b1f88e279b4729b54df390f28857b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" is_pre_election: true
I20260927 03:25:26.544798 14797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee79cdca4a84ecc87f064ddb5719f87" candidate_uuid: "76b1f88e279b4729b54df390f28857b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" is_pre_election: true
I20260927 03:25:26.544957 14797 raft_consensus.cc:2454] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76b1f88e279b4729b54df390f28857b0 in term 0.
I20260927 03:25:26.544957 14935 raft_consensus.cc:2454] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76b1f88e279b4729b54df390f28857b0 in term 0.
I20260927 03:25:26.545188 15003 leader_election.cc:302] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b1f88e279b4729b54df390f28857b0, d72e99c6079842e6a6ba2f37bdc136f9; no voters:
I20260927 03:25:26.545423 15128 raft_consensus.cc:2781] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:25:26.545514 15128 raft_consensus.cc:482] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:25:26.545531 15128 raft_consensus.cc:3037] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:26.546208 15128 raft_consensus.cc:504] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:26.546319 15128 leader_election.cc:288] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [CANDIDATE]: Term 1 election: Requested vote from peers 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143), d72e99c6079842e6a6ba2f37bdc136f9 (127.0.98.1:46281)
I20260927 03:25:26.546532 14935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee79cdca4a84ecc87f064ddb5719f87" candidate_uuid: "76b1f88e279b4729b54df390f28857b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba88939b2cb4e60b90af9e2275f57b0"
I20260927 03:25:26.546552 14797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee79cdca4a84ecc87f064ddb5719f87" candidate_uuid: "76b1f88e279b4729b54df390f28857b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72e99c6079842e6a6ba2f37bdc136f9"
I20260927 03:25:26.546603 14935 raft_consensus.cc:3037] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:26.546638 14797 raft_consensus.cc:3037] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:26.547435 14935 raft_consensus.cc:2454] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76b1f88e279b4729b54df390f28857b0 in term 1.
I20260927 03:25:26.547434 14797 raft_consensus.cc:2454] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76b1f88e279b4729b54df390f28857b0 in term 1.
I20260927 03:25:26.547603 15003 leader_election.cc:302] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba88939b2cb4e60b90af9e2275f57b0, 76b1f88e279b4729b54df390f28857b0; no voters:
I20260927 03:25:26.547693 15128 raft_consensus.cc:2781] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:25:26.547830 15128 raft_consensus.cc:686] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [term 1 LEADER]: Becoming Leader. State: Replica: 76b1f88e279b4729b54df390f28857b0, State: Running, Role: LEADER
I20260927 03:25:26.547924 15128 consensus_queue.cc:237] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:26.548792 14655 catalog_manager.cc:5495] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76b1f88e279b4729b54df390f28857b0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "76b1f88e279b4729b54df390f28857b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } health_report { overall_health: UNKNOWN } } }
I20260927 03:25:26.558162 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.556043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.0.98.3:34497)
0927 03:25:26.556102 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:26.556104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:26.556112 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:25:26.556147 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:26.556148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:26.556165 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:26.556221 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:25:26.556339 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.556796 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.557480 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:26.557661 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:26.557711 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:26.557852 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:25:26.557950 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:25:26.558014 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:26.558050 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 03:25:26.558070 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:25:26.559845 14797 raft_consensus.cc:1261] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Refusing update from remote peer 76b1f88e279b4729b54df390f28857b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:25:26.559883 14935 raft_consensus.cc:1261] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 1 FOLLOWER]: Refusing update from remote peer 76b1f88e279b4729b54df390f28857b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:25:26.560165 15128 consensus_queue.cc:1035] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:25:26.560302 15130 consensus_queue.cc:1035] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:25:26.562811 15134 mvcc.cc:205] Tried to move back new op lower bound from 7333804140787040256 to 7333804140741267456. Current Snapshot: MvccSnapshot[applied={T|T < 7333804140787040256 or (T in {7333804140787040256})}]
I20260927 03:25:26.563962 15135 mvcc.cc:205] Tried to move back new op lower bound from 7333804140787040256 to 7333804140741267456. Current Snapshot: MvccSnapshot[applied={T|T < 7333804140787040256}]
I20260927 03:25:26.564276 14654 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:25:26.564371 14654 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:25:26.565012 14654 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: Sending DeleteTablet for 3 replicas of tablet 1ee79cdca4a84ecc87f064ddb5719f87
I20260927 03:25:26.565316 14915 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ee79cdca4a84ecc87f064ddb5719f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:25:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51376
I20260927 03:25:26.565378 15047 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ee79cdca4a84ecc87f064ddb5719f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:25:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43490
I20260927 03:25:26.565521 14777 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ee79cdca4a84ecc87f064ddb5719f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:25:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40192
I20260927 03:25:26.565557 15143 tablet_replica.cc:330] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0: stopping tablet replica
I20260927 03:25:26.565604 14852 tablet_replica.cc:330] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9: stopping tablet replica
I20260927 03:25:26.565637 15143 raft_consensus.cc:2229] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:25:26.565646 14852 raft_consensus.cc:2229] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:25:26.565697 14852 raft_consensus.cc:2258] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:25:26.565724 15143 raft_consensus.cc:2258] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:25:26.566059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 14693
I20260927 03:25:26.566157 15143 ts_tablet_manager.cc:1905] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:25:26.567695 15142 tablet_replica.cc:330] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0: stopping tablet replica
I20260927 03:25:26.567831 15142 raft_consensus.cc:2229] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:25:26.567963 15142 raft_consensus.cc:2258] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:25:26.568409 15142 ts_tablet_manager.cc:1905] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:25:26.569948 15142 ts_tablet_manager.cc:1918] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:25:26.570055 15142 log.cc:1196] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/1ee79cdca4a84ecc87f064ddb5719f87
I20260927 03:25:26.570365 15142 ts_tablet_manager.cc:1939] T 1ee79cdca4a84ecc87f064ddb5719f87 P 2ba88939b2cb4e60b90af9e2275f57b0: Deleting consensus metadata
I20260927 03:25:26.570849 14642 catalog_manager.cc:4841] TS 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143): tablet 1ee79cdca4a84ecc87f064ddb5719f87 (table test-table [id=2d2baf1ad169476c8c9678023e9b0333]) successfully deleted
I20260927 03:25:26.570959 15143 ts_tablet_manager.cc:1918] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:25:26.571034 15143 log.cc:1196] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/1ee79cdca4a84ecc87f064ddb5719f87
I20260927 03:25:26.571282 15143 ts_tablet_manager.cc:1939] T 1ee79cdca4a84ecc87f064ddb5719f87 P 76b1f88e279b4729b54df390f28857b0: Deleting consensus metadata
I20260927 03:25:26.571656 14641 catalog_manager.cc:4841] TS 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497): tablet 1ee79cdca4a84ecc87f064ddb5719f87 (table test-table [id=2d2baf1ad169476c8c9678023e9b0333]) successfully deleted
W20260927 03:25:26.571661 14642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46281 (error 108)
I20260927 03:25:26.572005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 14848
I20260927 03:25:26.572618 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.572072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46281 (local address 127.0.0.1:40198)
0927 03:25:26.572545 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:26.572585 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450}
W20260927 03:25:26.572909 14642 catalog_manager.cc:4584] TS d72e99c6079842e6a6ba2f37bdc136f9 (127.0.98.1:46281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ee79cdca4a84ecc87f064ddb5719f87: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46281: connect: Connection refused (error 111)
I20260927 03:25:26.576836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 14984
I20260927 03:25:26.581388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 14622
I20260927 03:25:26.585880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35283
--webserver_interface=127.0.98.62
--webserver_port=36155
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:26.666546 15145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:26.666772 15145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:26.666795 15145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:26.668030 15145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:25:26.668081 15145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:26.668095 15145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:26.668107 15145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:25:26.668118 15145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:25:26.669672 15145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:26.669905 15145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:26.670143 15145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:25:26.675196 15145 server_base.cc:1034] running on GCE node
W20260927 03:25:26.675179 15150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:26.675207 15151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:26.675179 15153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:26.675639 15145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:26.675946 15145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:26.677107 15145 hybrid_clock.cc:648] HybridClock initialized: now 1790479526677092 us; error 30 us; skew 500 ppm
Sep 27 03:25:26 dist-test-slave-gw2v krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:26.679807 15145 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:25:26.680279 15145 webserver.cc:460] Webserver started at http://127.0.98.62:36155/ using document root <none> and password file <none>
I20260927 03:25:26.680469 15145 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:26.680526 15145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:26.681869 15145 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:25:26.682539 15160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:26.682688 15145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:25:26.682768 15145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "4d407a2dc4614e3abadc780570705a9e"
format_stamp: "Formatted at 2026-09-27 03:25:24 on dist-test-slave-gw2v"
server_key: "990a8319f06426d659cbc8fc9716f82d"
server_key_iv: "e2f0e5d6bbf8b93f76cf5a1db62fd013"
server_key_version: "encryptionkey@0"
I20260927 03:25:26.683156 15145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:26.693269 15145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:26.695472 15145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:26.695607 15145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:26.699584 15145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35283
I20260927 03:25:26.699616 15212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35283 every 8 connection(s)
I20260927 03:25:26.699972 15145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:25:26.700440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 15145
I20260927 03:25:26.700688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46281
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35283
--webserver_port=36891
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:25:26.701607 15213 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:25:26.702248 15213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: Bootstrap starting.
I20260927 03:25:26.710211 15213 log.cc:824] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:26.712472 15213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:25:26.712756 15213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e: Bootstrap complete.
I20260927 03:25:26.714802 15213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260927 03:25:26.715080 15213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d407a2dc4614e3abadc780570705a9e, State: Initialized, Role: FOLLOWER
I20260927 03:25:26.715235 15213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260927 03:25:26.715310 15213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:25:26.715358 15213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:25:26.715405 15213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 2 FOLLOWER]: Advancing to term 3
I20260927 03:25:26.716359 15213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260927 03:25:26.716450 15213 leader_election.cc:302] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d407a2dc4614e3abadc780570705a9e; no voters:
I20260927 03:25:26.716607 15213 leader_election.cc:288] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 03:25:26.716652 15217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 3 FOLLOWER]: Leader election won for term 3
I20260927 03:25:26.716770 15217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [term 3 LEADER]: Becoming Leader. State: Replica: 4d407a2dc4614e3abadc780570705a9e, State: Running, Role: LEADER
I20260927 03:25:26.716852 15217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } }
I20260927 03:25:26.716876 15213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:25:26.717168 15219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d407a2dc4614e3abadc780570705a9e. Latest consensus state: current_term: 3 leader_uuid: "4d407a2dc4614e3abadc780570705a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } } }
I20260927 03:25:26.717253 15219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:26.717178 15218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4d407a2dc4614e3abadc780570705a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d407a2dc4614e3abadc780570705a9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35283 } } }
I20260927 03:25:26.717294 15218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d407a2dc4614e3abadc780570705a9e [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:26.717429 15222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:25:26.718231 15222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:25:26.718349 15222 catalog_manager.cc:1259] Loaded cluster ID: ac2d5f874b464eae85c6c6300fe44183
I20260927 03:25:26.718389 15222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 03:25:26.718616 15222 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8ae51029ad google::LogMessage::Fail() at ??:0
@ 0x7f8ae510588a google::LogMessage::SendToLog() at ??:0
@ 0x7f8ae51024de google::LogMessage::Flush() at ??:0
@ 0x7f8ae5103a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8aed0230f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f8aed02c8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8ae6fd00d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8ae6fc5954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8ae7e296db start_thread at ??:0
@ 0x7f8ae3c0061f clone at ??:0
W20260927 03:25:26.797194 15215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:26.797430 15215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:26.797461 15215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:26.798679 15215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:26.798729 15215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:26.798759 15215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:25:26.800278 15215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36891
--tserver_master_addrs=127.0.98.62:35283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:26.800515 15215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:26.800757 15215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:26.805908 15249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:26.805905 15252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:26.805910 15250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:26.806131 15215 server_base.cc:1034] running on GCE node
I20260927 03:25:26.806471 15215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:26.806716 15215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:26.807873 15215 hybrid_clock.cc:648] HybridClock initialized: now 1790479526807881 us; error 51 us; skew 500 ppm
Sep 27 03:25:26 dist-test-slave-gw2v krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:26.810405 15215 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:25:26.811043 15215 webserver.cc:460] Webserver started at http://127.0.98.1:36891/ using document root <none> and password file <none>
I20260927 03:25:26.811313 15215 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:26.811388 15215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:26.812989 15215 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:25:26.813618 15259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:26.813791 15215 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:25:26.813952 15215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "d72e99c6079842e6a6ba2f37bdc136f9"
format_stamp: "Formatted at 2026-09-27 03:25:25 on dist-test-slave-gw2v"
server_key: "f748b207047a4e5a8dc2be9c136fdfbf"
server_key_iv: "e48c9b07ca9fe6f3cb95b174e92056b7"
server_key_version: "encryptionkey@0"
I20260927 03:25:26.814396 15215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:26.827533 15215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:26.830149 15215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:26.830310 15215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:26.830571 15215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:26.831148 15266 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:25:26.832764 15215 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:25:26.832842 15215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:25:26.832877 15215 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:25:26.833757 15215 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:25:26.833804 15215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:25:26.833915 15266 tablet_bootstrap.cc:492] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9: Bootstrap starting.
I20260927 03:25:26.840956 15215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46281
I20260927 03:25:26.841027 15373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46281 every 8 connection(s)
I20260927 03:25:26.841418 15215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:25:26.844178 15266 log.cc:824] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:26.845701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 15215
I20260927 03:25:26.845991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46143
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35283
--webserver_port=36633
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:25:26.846589 15266 tablet_bootstrap.cc:492] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:25:26.846863 15266 tablet_bootstrap.cc:492] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9: Bootstrap complete.
I20260927 03:25:26.847703 15266 ts_tablet_manager.cc:1397] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
I20260927 03:25:26.848767 15266 raft_consensus.cc:348] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:26.849004 15266 raft_consensus.cc:729] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d72e99c6079842e6a6ba2f37bdc136f9, State: Initialized, Role: FOLLOWER
I20260927 03:25:26.849133 15266 consensus_queue.cc:260] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:26.849416 15266 ts_tablet_manager.cc:1428] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260927 03:25:26.914507 15376 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:25:26.842093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:54027)
0927 03:25:26.842274 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:26.842320 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:26.843021 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:26.914325 (+ 71304us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
W20260927 03:25:26.914742 15374 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35283 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35283: Client connection negotiation failed: client connection to 127.0.98.62:35283: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:25:26.915141 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.915032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:41957)
0927 03:25:26.915080 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:26.915120 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 03:25:26.915472 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.915388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:36631)
0927 03:25:26.915432 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:26.915456 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 03:25:26.915591 15374 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:25:26.930306 15378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:26.930528 15378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:26.930560 15378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:26.931794 15378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:26.931846 15378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:26.931883 15378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:25:26.933382 15378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36633
--tserver_master_addrs=127.0.98.62:35283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:26.933620 15378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:26.933835 15378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:26.938910 15387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:26.938796 15385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:26.938777 15384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:26.939109 15378 server_base.cc:1034] running on GCE node
I20260927 03:25:26.939339 15378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:26.939572 15378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:26.940800 15378 hybrid_clock.cc:648] HybridClock initialized: now 1790479526940786 us; error 29 us; skew 500 ppm
Sep 27 03:25:26 dist-test-slave-gw2v krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:26.943500 15378 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:25:26.943933 15378 webserver.cc:460] Webserver started at http://127.0.98.2:36633/ using document root <none> and password file <none>
I20260927 03:25:26.944128 15378 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:26.944172 15378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:26.945502 15378 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:25:26.946115 15394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:26.946281 15378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:26.946391 15378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "2ba88939b2cb4e60b90af9e2275f57b0"
format_stamp: "Formatted at 2026-09-27 03:25:25 on dist-test-slave-gw2v"
server_key: "53b71e254aed891b4cbc08b293190836"
server_key_iv: "2af91cc0120951223f351425fdfa9776"
server_key_version: "encryptionkey@0"
I20260927 03:25:26.946725 15378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:26.974792 15378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:26.976915 15378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:26.977032 15378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:26.977267 15378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:26.977613 15378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:26.977653 15378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:26.977674 15378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:26.977685 15378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:26.983299 15378 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46143
I20260927 03:25:26.983353 15507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46143 every 8 connection(s)
I20260927 03:25:26.983762 15378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:25:26.984861 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.984464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:34113)
0927 03:25:26.984634 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:26.984708 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
W20260927 03:25:26.985327 15508 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35283 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35283: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
I20260927 03:25:26.985600 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.985510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:44787)
0927 03:25:26.985551 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:26.985582 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 03:25:26.986044 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:26.985935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:34213)
0927 03:25:26.985997 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:26.986025 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 03:25:26.986162 15508 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:25:26.993674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 15378
I20260927 03:25:26.993916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34497
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35283
--webserver_port=36319
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:27.088392 15511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:27.088604 15511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:27.088627 15511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:27.089780 15511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:27.089821 15511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:27.089847 15511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:25:27.091313 15511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36319
--tserver_master_addrs=127.0.98.62:35283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:27.091557 15511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:27.091771 15511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:25:27.096755 15511 server_base.cc:1034] running on GCE node
W20260927 03:25:27.096624 15516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:27.096627 15517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:27.096668 15519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:27.097175 15511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:27.097424 15511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:27.098562 15511 hybrid_clock.cc:648] HybridClock initialized: now 1790479527098548 us; error 29 us; skew 500 ppm
Sep 27 03:25:27 dist-test-slave-gw2v krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:27.101086 15511 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:25:27.101526 15511 webserver.cc:460] Webserver started at http://127.0.98.3:36319/ using document root <none> and password file <none>
I20260927 03:25:27.101734 15511 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:27.101781 15511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:27.103120 15511 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:25:27.103806 15526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:27.103963 15511 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:25:27.104046 15511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "76b1f88e279b4729b54df390f28857b0"
format_stamp: "Formatted at 2026-09-27 03:25:25 on dist-test-slave-gw2v"
server_key: "f1a472b8b065b0ad68245f2e9c322349"
server_key_iv: "7cf319b8ffb10b8deb1da48d84ba85da"
server_key_version: "encryptionkey@0"
I20260927 03:25:27.104398 15511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:27.121413 15511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:27.123544 15511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:27.123680 15511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:27.123894 15511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:27.124195 15511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:27.124230 15511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:27.124259 15511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:27.124284 15511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:27.129551 15511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34497
I20260927 03:25:27.129614 15639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34497 every 8 connection(s)
I20260927 03:25:27.129944 15511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:25:27.130957 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:27.130579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:40283)
0927 03:25:27.130713 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:27.130799 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
W20260927 03:25:27.131343 15640 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35283 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35283: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
I20260927 03:25:27.131579 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:27.131503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:44821)
0927 03:25:27.131540 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:27.131564 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 03:25:27.131803 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:27.131743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:57971)
0927 03:25:27.131769 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:27.131790 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260927 03:25:27.131887 15640 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:25:27.138979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 15511
I20260927 03:25:27.916303 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:27.916028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:59667)
0927 03:25:27.916183 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:27.916232 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260927 03:25:27.986760 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:27.986509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:48871)
0927 03:25:27.986648 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:27.986693 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260927 03:25:27.990252 15646 raft_consensus.cc:482] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:25:27.990376 15646 raft_consensus.cc:504] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:27.990691 15646 leader_election.cc:288] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143), 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497)
Sep 27 03:25:27 dist-test-slave-gw2v krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 03:25:27 dist-test-slave-gw2v krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 03:25:28.001539 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:27.990931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34497 (local address 127.0.98.1:38527)
0927 03:25:27.991033 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:27.991042 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:27.991091 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:27.991899 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:27.991902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:27.992269 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:27.992453 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:27.992459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:27.993613 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:27.993614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:27.994443 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:27.994446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:27.995073 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:27.995074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:27.995186 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:27.995747 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:27.995752 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:27.997153 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:27.999529 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:27.999536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:27.999539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:27.999946 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:28.000296 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:28.000297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:28.000297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:28.000476 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:28.000966 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:28.000970 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:28.001131 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 03:25:28.001246 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 03:25:28.001428 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":96,"thread_start_us":51,"threads_started":1}
I20260927 03:25:28.001574 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:27.991050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38527 (local address 127.0.98.3:34497)
0927 03:25:27.991200 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:27.991211 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:27.991230 (+ 19us) server_negotiation.cc:407] Connection header received
0927 03:25:27.991651 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:27.991654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:27.991719 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:27.991840 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:27.992558 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:27.993531 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:27.994541 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:27.994992 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:27.995093 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:27.997514 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:27.997517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:27.997518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:27.997527 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:27.999423 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:28.000040 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:28.000043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:28.000044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:28.000198 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:28.000554 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:28.000555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:28.000555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:28.000886 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:28.000974 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:28.001328 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 03:25:28.001469 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260927 03:25:28.001585 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:27.991143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42765 (local address 127.0.98.2:46143)
0927 03:25:27.991255 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:27.991263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:27.991291 (+ 28us) server_negotiation.cc:407] Connection header received
0927 03:25:27.991702 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:27.991704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:27.991763 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:27.991894 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:27.992576 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:27.993626 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:27.994530 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:27.994992 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:27.995096 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:27.997497 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:27.997502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:27.997502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:27.997514 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:27.999449 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:28.000040 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:28.000043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:28.000044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:28.000198 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:28.000554 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:28.000555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:28.000555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:28.000880 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:28.000974 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:28.001328 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 03:25:28.001469 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":42,"threads_started":1}
I20260927 03:25:28.001596 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:27.990931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46143 (local address 127.0.98.1:42765)
0927 03:25:27.990986 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:27.990998 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:27.991071 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:27.991901 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:27.991904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:27.992276 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:27.992453 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:27.992459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:27.993721 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:27.993722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:27.994442 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:27.994445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:27.995076 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:27.995077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:27.995187 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:27.995714 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:27.995721 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:27.997403 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:27.999529 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:27.999536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:27.999539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:27.999946 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:28.000296 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:28.000297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:28.000297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:28.000476 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:28.000963 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:28.000970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:28.001131 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 03:25:28.001246 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 03:25:28.001495 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":119}
W20260927 03:25:28.002918 15263 leader_election.cc:341] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143): Not found: Tablet not found: 1ee79cdca4a84ecc87f064ddb5719f87
W20260927 03:25:28.003019 15262 leader_election.cc:341] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497): Not found: Tablet not found: 1ee79cdca4a84ecc87f064ddb5719f87
I20260927 03:25:28.003085 15262 leader_election.cc:302] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d72e99c6079842e6a6ba2f37bdc136f9; no voters: 2ba88939b2cb4e60b90af9e2275f57b0, 76b1f88e279b4729b54df390f28857b0
I20260927 03:25:28.003206 15646 raft_consensus.cc:2726] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:25:28.132434 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:28.132202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:33399)
0927 03:25:28.132341 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:28.132385 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260927 03:25:28.917127 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:28.916895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:58657)
0927 03:25:28.917034 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:28.917079 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260927 03:25:28.987545 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:28.987266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:50401)
0927 03:25:28.987429 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:28.987479 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":66,"threads_started":1}
I20260927 03:25:29.133114 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:29.132890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:46357)
0927 03:25:29.133022 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:29.133071 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260927 03:25:29.917906 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:29.917639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:46229)
0927 03:25:29.917805 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:29.917850 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
I20260927 03:25:29.972532 15655 raft_consensus.cc:482] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:25:29.972645 15655 raft_consensus.cc:504] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:29.972781 15655 leader_election.cc:288] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143), 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497)
W20260927 03:25:29.973182 15262 leader_election.cc:341] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497): Not found: Tablet not found: 1ee79cdca4a84ecc87f064ddb5719f87
W20260927 03:25:29.973258 15263 leader_election.cc:341] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143): Not found: Tablet not found: 1ee79cdca4a84ecc87f064ddb5719f87
I20260927 03:25:29.973326 15263 leader_election.cc:302] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d72e99c6079842e6a6ba2f37bdc136f9; no voters: 2ba88939b2cb4e60b90af9e2275f57b0, 76b1f88e279b4729b54df390f28857b0
I20260927 03:25:29.973393 15655 raft_consensus.cc:2726] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:25:29.988273 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:29.988054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:46243)
0927 03:25:29.988177 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:29.988226 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260927 03:25:30.133786 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:30.133553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:33351)
0927 03:25:30.133699 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:30.133744 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260927 03:25:30.918721 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:30.918434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:52739)
0927 03:25:30.918625 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:30.918673 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":119,"threads_started":1}
I20260927 03:25:30.988984 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:30.988736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:44565)
0927 03:25:30.988890 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:30.988936 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
I20260927 03:25:31.134573 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:31.134308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:38409)
0927 03:25:31.134482 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:31.134523 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
I20260927 03:25:31.919528 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:31.919236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:51587)
0927 03:25:31.919403 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:31.919455 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":67,"threads_started":1}
I20260927 03:25:31.989725 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:31.989503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:48133)
0927 03:25:31.989642 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:31.989682 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260927 03:25:32.135285 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:32.135063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:34507)
0927 03:25:32.135195 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:32.135237 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260927 03:25:32.893321 15664 raft_consensus.cc:482] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:25:32.893424 15664 raft_consensus.cc:504] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:32.893565 15664 leader_election.cc:288] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143), 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497)
W20260927 03:25:32.893988 15263 leader_election.cc:341] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143): Not found: Tablet not found: 1ee79cdca4a84ecc87f064ddb5719f87
W20260927 03:25:32.894074 15262 leader_election.cc:341] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497): Not found: Tablet not found: 1ee79cdca4a84ecc87f064ddb5719f87
I20260927 03:25:32.894141 15262 leader_election.cc:302] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d72e99c6079842e6a6ba2f37bdc136f9; no voters: 2ba88939b2cb4e60b90af9e2275f57b0, 76b1f88e279b4729b54df390f28857b0
I20260927 03:25:32.894202 15664 raft_consensus.cc:2726] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:25:32.920269 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:32.920036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:49379)
0927 03:25:32.920175 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:32.920221 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260927 03:25:32.990470 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:32.990240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:46599)
0927 03:25:32.990387 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:32.990424 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260927 03:25:33.136036 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:33.135805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:34333)
0927 03:25:33.135947 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:33.135992 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260927 03:25:33.921069 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:33.920811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:45447)
0927 03:25:33.920978 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:33.921022 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":82,"threads_started":1}
I20260927 03:25:33.991348 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:33.991109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:52749)
0927 03:25:33.991254 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:33.991300 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260927 03:25:34.136751 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:34.136522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:51621)
0927 03:25:34.136657 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:34.136704 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260927 03:25:34.921732 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:34.921539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:56163)
0927 03:25:34.921652 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:34.921684 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260927 03:25:34.992141 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:34.991904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:54577)
0927 03:25:34.992041 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:34.992088 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260927 03:25:35.137476 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:35.137242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:57429)
0927 03:25:35.137386 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:35.137429 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260927 03:25:35.300516 15674 raft_consensus.cc:482] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:25:35.300624 15674 raft_consensus.cc:504] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:35.300765 15674 leader_election.cc:288] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143), 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497)
W20260927 03:25:35.301178 15263 leader_election.cc:341] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143): Not found: Tablet not found: 1ee79cdca4a84ecc87f064ddb5719f87
W20260927 03:25:35.301258 15262 leader_election.cc:341] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497): Not found: Tablet not found: 1ee79cdca4a84ecc87f064ddb5719f87
I20260927 03:25:35.301328 15262 leader_election.cc:302] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d72e99c6079842e6a6ba2f37bdc136f9; no voters: 2ba88939b2cb4e60b90af9e2275f57b0, 76b1f88e279b4729b54df390f28857b0
I20260927 03:25:35.301422 15674 raft_consensus.cc:2726] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:25:35.922462 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:35.922201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:47507)
0927 03:25:35.922353 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:35.922410 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260927 03:25:35.992799 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:35.992566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:50889)
0927 03:25:35.992704 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:35.992743 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260927 03:25:36.138163 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:36.137927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:39495)
0927 03:25:36.138078 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:36.138118 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
I20260927 03:25:36.923132 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:36.922936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:53441)
0927 03:25:36.923049 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:36.923084 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260927 03:25:36.993487 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:36.993274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:43427)
0927 03:25:36.993399 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:36.993441 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260927 03:25:37.138895 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:37.138623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:55809)
0927 03:25:37.138773 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:37.138847 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260927 03:25:37.923754 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:37.923530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:48667)
0927 03:25:37.923667 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:37.923698 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":59,"threads_started":1}
I20260927 03:25:37.994174 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:37.993966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:45241)
0927 03:25:37.994086 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:37.994126 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260927 03:25:38.139569 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:38.139347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:36513)
0927 03:25:38.139485 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:38.139524 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260927 03:25:38.924420 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:38.924194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:59067)
0927 03:25:38.924327 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:38.924375 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260927 03:25:38.994879 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:38.994602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:36769)
0927 03:25:38.994751 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:38.994796 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260927 03:25:39.140244 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:39.140011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:55889)
0927 03:25:39.140154 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:39.140194 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260927 03:25:39.925089 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:39.924837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:41709)
0927 03:25:39.924968 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:39.925041 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260927 03:25:39.995533 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:39.995304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:57421)
0927 03:25:39.995448 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:39.995488 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260927 03:25:40.140990 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:40.140751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:35483)
0927 03:25:40.140905 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:40.140943 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260927 03:25:40.925738 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:40.925515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:42157)
0927 03:25:40.925653 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:40.925691 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260927 03:25:40.996254 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:40.996031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:42861)
0927 03:25:40.996169 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:40.996209 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260927 03:25:41.141693 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:41.141454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:43371)
0927 03:25:41.141595 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:41.141642 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260927 03:25:41.363824 15693 raft_consensus.cc:482] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:25:41.363919 15693 raft_consensus.cc:504] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba88939b2cb4e60b90af9e2275f57b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46143 } } peers { permanent_uuid: "76b1f88e279b4729b54df390f28857b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34497 } } peers { permanent_uuid: "d72e99c6079842e6a6ba2f37bdc136f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46281 } }
I20260927 03:25:41.364076 15693 leader_election.cc:288] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143), 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497)
W20260927 03:25:41.364387 15263 leader_election.cc:341] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ba88939b2cb4e60b90af9e2275f57b0 (127.0.98.2:46143): Not found: Tablet not found: 1ee79cdca4a84ecc87f064ddb5719f87
W20260927 03:25:41.364468 15262 leader_election.cc:341] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 76b1f88e279b4729b54df390f28857b0 (127.0.98.3:34497): Not found: Tablet not found: 1ee79cdca4a84ecc87f064ddb5719f87
I20260927 03:25:41.364511 15262 leader_election.cc:302] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d72e99c6079842e6a6ba2f37bdc136f9; no voters: 2ba88939b2cb4e60b90af9e2275f57b0, 76b1f88e279b4729b54df390f28857b0
I20260927 03:25:41.364599 15693 raft_consensus.cc:2726] T 1ee79cdca4a84ecc87f064ddb5719f87 P d72e99c6079842e6a6ba2f37bdc136f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:25:41.926502 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:41.926241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.1:34821)
0927 03:25:41.926415 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:41.926453 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":69,"threads_started":1}
I20260927 03:25:41.997005 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:41.996783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.2:43075)
0927 03:25:41.996915 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:41.996956 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260927 03:25:42.142557 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:42.142312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35283 (local address 127.0.98.3:41063)
0927 03:25:42.142461 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:42.142505 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260927 03:25:42.145395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 15215
I20260927 03:25:42.149461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 15378
I20260927 03:25:42.153716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 15511
2026-09-27T03:25:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18324 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:25:42 dist-test-slave-gw2v krb5kdc[15699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:25:42 dist-test-slave-gw2v krb5kdc[15699](info): set up 2 sockets
Sep 27 03:25:42 dist-test-slave-gw2v krb5kdc[15699](info): commencing operation
krb5kdc: starting...
W20260927 03:25:43.102715 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:25:43 dist-test-slave-gw2v krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:25:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:25:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:25:43.217542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34447
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:43.296955 15712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:43.297178 15712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:43.297209 15712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:43.298385 15712 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 03:25:43.298430 15712 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 03:25:43.298446 15712 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 03:25:43.298460 15712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:25:43.298475 15712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:43.298493 15712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:43.298517 15712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:25:43.298535 15712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:25:43.300130 15712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:43.300400 15712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:43.300613 15712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:43.305486 15717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:43.305505 15718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:43.305505 15720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:43.305737 15712 server_base.cc:1034] running on GCE node
I20260927 03:25:43.306002 15712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:43.306295 15712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:43.307435 15712 hybrid_clock.cc:648] HybridClock initialized: now 1790479543307416 us; error 38 us; skew 500 ppm
Sep 27 03:25:43 dist-test-slave-gw2v krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:43.309804 15712 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:25:43.310268 15712 webserver.cc:460] Webserver started at http://127.0.98.62:42767/ using document root <none> and password file <none>
I20260927 03:25:43.310474 15712 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:43.310518 15712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:43.310643 15712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:25:43.311733 15712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "c8c0a65633f045f499723665fdc34fbc"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "c14cf596fbaf9c94d82822fba04e386c"
server_key_iv: "f3b1c923280cef4ed6699a0f10e7bc98"
server_key_version: "encryptionkey@0"
I20260927 03:25:43.312067 15712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "c8c0a65633f045f499723665fdc34fbc"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "c14cf596fbaf9c94d82822fba04e386c"
server_key_iv: "f3b1c923280cef4ed6699a0f10e7bc98"
server_key_version: "encryptionkey@0"
I20260927 03:25:43.313293 15712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:25:43.313947 15727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:43.314102 15712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:25:43.314186 15712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "c8c0a65633f045f499723665fdc34fbc"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "c14cf596fbaf9c94d82822fba04e386c"
server_key_iv: "f3b1c923280cef4ed6699a0f10e7bc98"
server_key_version: "encryptionkey@0"
I20260927 03:25:43.314241 15712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:43.326958 15712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:43.329094 15712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:43.329206 15712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:43.332911 15712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34447
I20260927 03:25:43.332963 15779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34447 every 8 connection(s)
I20260927 03:25:43.333271 15712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:25:43.334098 15780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:43.336410 15780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Bootstrap starting.
I20260927 03:25:43.336970 15780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:43.337225 15780 log.cc:824] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:43.337915 15780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: No bootstrap required, opened a new log
I20260927 03:25:43.339604 15780 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } }
I20260927 03:25:43.339752 15780 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:43.339784 15780 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c0a65633f045f499723665fdc34fbc, State: Initialized, Role: FOLLOWER
I20260927 03:25:43.339926 15780 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } }
I20260927 03:25:43.340003 15780 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:25:43.340044 15780 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:25:43.340088 15780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:43.340607 15780 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } }
I20260927 03:25:43.340699 15780 leader_election.cc:302] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8c0a65633f045f499723665fdc34fbc; no voters:
I20260927 03:25:43.340862 15780 leader_election.cc:288] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:25:43.340943 15783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:25:43.341101 15780 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:25:43.341149 15783 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 1 LEADER]: Becoming Leader. State: Replica: c8c0a65633f045f499723665fdc34fbc, State: Running, Role: LEADER
I20260927 03:25:43.341276 15783 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } }
I20260927 03:25:43.341634 15784 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8c0a65633f045f499723665fdc34fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } } }
I20260927 03:25:43.341631 15785 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8c0a65633f045f499723665fdc34fbc. Latest consensus state: current_term: 1 leader_uuid: "c8c0a65633f045f499723665fdc34fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } } }
I20260927 03:25:43.341727 15784 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:43.342036 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 15712
I20260927 03:25:43.342123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:25:43.341982 15789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:25:43.342335 392 external_mini_cluster.cc:1428] Setting key eb66dfbcd185b6bef20208d18a641246
I20260927 03:25:43.341733 15785 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:25:43.342782 15789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:25:43.344729 15789 catalog_manager.cc:1347] Generated new cluster ID: a6923a81258848978b234a924ee2b699
I20260927 03:25:43.344774 15789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:25:43.350603 15789 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:25:43.351384 15789 catalog_manager.cc:1504] Loading token signing keys...
Sep 27 03:25:43 dist-test-slave-gw2v krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:43.357386 15789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Generated new TSK 0
I20260927 03:25:43.357533 15789 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:25:43.360549 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.343077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.0.98.62:34447)
0927 03:25:43.350420 (+ 7343us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:43.350430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:43.350470 (+ 40us) server_negotiation.cc:407] Connection header received
0927 03:25:43.350603 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:43.350612 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:43.350677 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:43.350912 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:43.351661 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.353129 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.354124 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.354779 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.354926 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.357573 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:43.357577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:43.357578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:43.357589 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:43.359125 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:43.359507 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:43.359509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:43.359509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:43.359593 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:43.359771 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:43.359771 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:43.359772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:43.359990 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:43.360093 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:43.360224 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 03:25:43.360332 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7291,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:25:43.389307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34447
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:43.469720 15806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:43.469928 15806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:43.469957 15806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:43.471132 15806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:43.471171 15806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:43.471206 15806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:25:43.472642 15806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:43.472869 15806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:43.473074 15806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:43.477890 15812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:43.477890 15811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:43.477995 15806 server_base.cc:1034] running on GCE node
W20260927 03:25:43.477964 15814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:43.478323 15806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:43.478581 15806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:43.479722 15806 hybrid_clock.cc:648] HybridClock initialized: now 1790479543479708 us; error 27 us; skew 500 ppm
Sep 27 03:25:43 dist-test-slave-gw2v krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:43.482112 15806 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:25:43.482523 15806 webserver.cc:460] Webserver started at http://127.0.98.1:38795/ using document root <none> and password file <none>
I20260927 03:25:43.482735 15806 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:43.482782 15806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:43.482932 15806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:25:43.483971 15806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "344fa21d06a94b5a9ed279793f932e22"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "530e3e60fce0c5342da60805f7f09bdb"
server_key_iv: "5c6c801959963757230cd9c84465a6bb"
server_key_version: "encryptionkey@0"
I20260927 03:25:43.484290 15806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "344fa21d06a94b5a9ed279793f932e22"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "530e3e60fce0c5342da60805f7f09bdb"
server_key_iv: "5c6c801959963757230cd9c84465a6bb"
server_key_version: "encryptionkey@0"
I20260927 03:25:43.485549 15806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:25:43.486196 15821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:43.486384 15806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:25:43.486461 15806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "344fa21d06a94b5a9ed279793f932e22"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "530e3e60fce0c5342da60805f7f09bdb"
server_key_iv: "5c6c801959963757230cd9c84465a6bb"
server_key_version: "encryptionkey@0"
I20260927 03:25:43.486518 15806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:43.495805 15806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:43.497543 15806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:43.497673 15806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:43.497889 15806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:43.498170 15806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:43.498201 15806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:43.498229 15806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:43.498250 15806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:43.503571 15806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42967
I20260927 03:25:43.503631 15934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42967 every 8 connection(s)
I20260927 03:25:43.503925 15806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:25:43.504316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 15806
I20260927 03:25:43.504424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:25:43.504611 392 external_mini_cluster.cc:1428] Setting key 7924144ad6caef1e078c222fdddab1f1
Sep 27 03:25:43 dist-test-slave-gw2v krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 03:25:43.513906 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.504720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36511 (local address 127.0.98.62:34447)
0927 03:25:43.504777 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:43.504779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:43.504993 (+ 214us) server_negotiation.cc:407] Connection header received
0927 03:25:43.505597 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:43.505598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:43.505609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:43.505659 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:43.506665 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.507217 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.508604 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.508802 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.508879 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.511179 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:43.511180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:43.511181 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:43.511187 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:43.512552 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:43.513038 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:43.513039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:43.513039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:43.513095 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:43.513361 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:43.513361 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:43.513361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:43.513463 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:43.513509 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:43.513765 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 03:25:43.513837 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:25:43.514091 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.504702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:36511)
0927 03:25:43.504881 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:43.504929 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:43.505503 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:43.505725 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:43.505728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:43.506287 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:43.506574 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.506581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.507301 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.507302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.508524 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.508527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.508888 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.508889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.509040 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.509552 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:43.509560 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:43.511070 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:43.512651 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:43.512660 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:43.512662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:43.512959 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:43.513159 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:43.513160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:43.513161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:43.513302 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:43.513513 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:43.513517 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:43.513662 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 03:25:43.513750 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 03:25:43.513882 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260927 03:25:43.514664 15935 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34447
I20260927 03:25:43.514752 15935 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:43.514957 15935 heartbeater.cc:511] Master 127.0.98.62:34447 requested a full tablet report, sending...
I20260927 03:25:43.515489 15744 ts_manager.cc:194] Registered new tserver with Master: 344fa21d06a94b5a9ed279793f932e22 (127.0.98.1:42967)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 03:25:43.516238 15744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:25:43.530305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34447
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:43.610030 15940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:43.610236 15940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:43.610268 15940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:43.611452 15940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:43.611495 15940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:43.611528 15940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:25:43.612923 15940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:43.613148 15940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:43.613356 15940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:43.618119 15946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:43.618134 15945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:43.618130 15948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:43.618178 15940 server_base.cc:1034] running on GCE node
I20260927 03:25:43.618482 15940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:43.618705 15940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:43.619838 15940 hybrid_clock.cc:648] HybridClock initialized: now 1790479543619825 us; error 28 us; skew 500 ppm
Sep 27 03:25:43 dist-test-slave-gw2v krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:43.622195 15940 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:25:43.622650 15940 webserver.cc:460] Webserver started at http://127.0.98.2:42145/ using document root <none> and password file <none>
I20260927 03:25:43.622884 15940 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:43.622933 15940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:43.623060 15940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:25:43.624078 15940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "66b09b2401454b29b307204e88c68c31"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "469924b985dc159629fc406182f0331e"
server_key_iv: "06a1f6a539187217a7551ae829475bb7"
server_key_version: "encryptionkey@0"
I20260927 03:25:43.624419 15940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "66b09b2401454b29b307204e88c68c31"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "469924b985dc159629fc406182f0331e"
server_key_iv: "06a1f6a539187217a7551ae829475bb7"
server_key_version: "encryptionkey@0"
I20260927 03:25:43.625747 15940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:25:43.626483 15955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:43.626644 15940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:25:43.626727 15940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "66b09b2401454b29b307204e88c68c31"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "469924b985dc159629fc406182f0331e"
server_key_iv: "06a1f6a539187217a7551ae829475bb7"
server_key_version: "encryptionkey@0"
I20260927 03:25:43.626796 15940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:43.633503 15940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:43.635334 15940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:43.635466 15940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:43.635669 15940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:43.635963 15940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:43.635996 15940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:43.636024 15940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:43.636042 15940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:43.641556 15940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45867
I20260927 03:25:43.641592 16068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45867 every 8 connection(s)
I20260927 03:25:43.641957 15940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:25:43.645136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 15940
I20260927 03:25:43.645246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:25:43.645428 392 external_mini_cluster.cc:1428] Setting key 6cb30e93aff63fbc03d66a4ba8da1934
Sep 27 03:25:43 dist-test-slave-gw2v krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:43.651572 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.642607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55845 (local address 127.0.98.62:34447)
0927 03:25:43.642650 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:43.642652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:43.642851 (+ 199us) server_negotiation.cc:407] Connection header received
0927 03:25:43.643411 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:43.643412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:43.643423 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:43.643468 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:43.644384 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.644879 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.646204 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.646391 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.646441 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.648950 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:43.648952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:43.648952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:43.648958 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:43.650208 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:43.650659 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:43.650660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:43.650660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:43.650703 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:43.650963 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:43.650964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:43.650964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:43.651062 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:43.651110 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:43.651444 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 03:25:43.651507 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:25:43.651820 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.642592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:55845)
0927 03:25:43.642724 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:43.642759 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:43.643326 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:43.643530 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:43.643533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:43.644069 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:43.644264 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.644269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.644957 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.644958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.646124 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.646128 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.646468 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.646469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.646607 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.647114 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:43.647124 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:43.648853 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:43.650284 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:43.650291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:43.650293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:43.650591 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:43.650742 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:43.650743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:43.650744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:43.650905 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:43.651120 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:43.651126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:43.651296 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 03:25:43.651424 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 03:25:43.651590 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260927 03:25:43.652257 16069 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34447
I20260927 03:25:43.652347 16069 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:43.652514 16069 heartbeater.cc:511] Master 127.0.98.62:34447 requested a full tablet report, sending...
I20260927 03:25:43.652905 15744 ts_manager.cc:194] Registered new tserver with Master: 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867)
I20260927 03:25:43.653410 15744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55845
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:25:43.673941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34447
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:25:43.754395 16074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:25:43.754631 16074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:25:43.754658 16074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:25:43.755890 16074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:25:43.755934 16074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:25:43.755967 16074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:25:43.757434 16074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:25:43.757627 16074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:25:43.757803 16074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:25:43.762526 16079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:43.762535 16080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:25:43.762535 16082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:25:43.762745 16074 server_base.cc:1034] running on GCE node
I20260927 03:25:43.762964 16074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:25:43.763183 16074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:25:43.764324 16074 hybrid_clock.cc:648] HybridClock initialized: now 1790479543764308 us; error 31 us; skew 500 ppm
Sep 27 03:25:43 dist-test-slave-gw2v krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:25:43.766641 16074 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:25:43.767087 16074 webserver.cc:460] Webserver started at http://127.0.98.3:44131/ using document root <none> and password file <none>
I20260927 03:25:43.767293 16074 fs_manager.cc:359] Metadata directory not provided
I20260927 03:25:43.767347 16074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:25:43.767469 16074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:25:43.768477 16074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "62e14b49b72140008596d3632165bd88"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "d8d89862e5b6c60f2f9ffcb3dbd84b67"
server_key_iv: "7c1b9319d5758c0371ca9d523b819b9f"
server_key_version: "encryptionkey@0"
I20260927 03:25:43.768783 16074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "62e14b49b72140008596d3632165bd88"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "d8d89862e5b6c60f2f9ffcb3dbd84b67"
server_key_iv: "7c1b9319d5758c0371ca9d523b819b9f"
server_key_version: "encryptionkey@0"
I20260927 03:25:43.769953 16074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:25:43.770589 16089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:43.770738 16074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:25:43.770833 16074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "62e14b49b72140008596d3632165bd88"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "d8d89862e5b6c60f2f9ffcb3dbd84b67"
server_key_iv: "7c1b9319d5758c0371ca9d523b819b9f"
server_key_version: "encryptionkey@0"
I20260927 03:25:43.770898 16074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:25:43.790258 16074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:25:43.792507 16074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:25:43.792649 16074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:25:43.792869 16074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:25:43.793157 16074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:25:43.793190 16074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:43.793218 16074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:25:43.793241 16074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:25:43.798786 16074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37929
I20260927 03:25:43.798875 16202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37929 every 8 connection(s)
I20260927 03:25:43.799166 16074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:25:43 dist-test-slave-gw2v krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:25:43.807490 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.799847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33357 (local address 127.0.98.62:34447)
0927 03:25:43.799897 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:43.799899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:43.800084 (+ 185us) server_negotiation.cc:407] Connection header received
0927 03:25:43.800544 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:43.800545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:43.800553 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:43.800617 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:43.801481 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.801983 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.802962 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.803141 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.803186 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.805210 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:43.805211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:43.805212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:43.805219 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:43.806340 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:43.806691 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:43.806692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:43.806692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:43.806742 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:43.806947 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:43.806948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:43.806948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:43.807055 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:43.807105 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:43.807358 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 03:25:43.807421 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:25:43.807682 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.799814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:33357)
0927 03:25:43.799964 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:43.800007 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:43.800471 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:43.800675 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:43.800678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:43.801215 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:25:43.801416 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.801420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.802046 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.802047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.802896 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.802898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.803195 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.803196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.803295 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.803798 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:25:43.803809 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:25:43.805117 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:25:43.806415 (+ 1298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:43.806421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:43.806423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:43.806629 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:43.806786 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:25:43.806787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:25:43.806787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:25:43.806902 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:25:43.807109 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:25:43.807114 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:25:43.807238 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 03:25:43.807339 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 03:25:43.807463 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260927 03:25:43.808106 16203 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34447
I20260927 03:25:43.808199 16203 heartbeater.cc:464] Registering TS with master...
I20260927 03:25:43.808382 16203 heartbeater.cc:511] Master 127.0.98.62:34447 requested a full tablet report, sending...
I20260927 03:25:43.808697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 16074
I20260927 03:25:43.808751 15744 ts_manager.cc:194] Registered new tserver with Master: 62e14b49b72140008596d3632165bd88 (127.0.98.3:37929)
I20260927 03:25:43.808820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:25:43.808998 392 external_mini_cluster.cc:1428] Setting key f2f2b248cf9cec2505b5d699f1f2614d
I20260927 03:25:43.809194 15744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33357
I20260927 03:25:43.809849 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:25:43.816864 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.811144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51488 (local address 127.0.98.62:34447)
0927 03:25:43.811187 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:43.811189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:43.811318 (+ 129us) server_negotiation.cc:407] Connection header received
0927 03:25:43.811348 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:43.811349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:43.811357 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:43.811389 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:25:43.811889 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.812363 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.813067 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.813247 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.813316 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.814478 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:25:43.814479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:25:43.814479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:25:43.814486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:25:43.815871 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:43.816184 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:43.816184 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:43.816185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:43.816274 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:25:43.816434 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:25:43.816434 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:25:43.816434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:25:43.816570 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:25:43.816641 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:43.816743 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 03:25:43.816800 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:25:43.818672 15743 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:25:43.820258 15743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:25:43.826947 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.822219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37929 (local address 127.0.0.1:51122)
0927 03:25:43.822387 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:43.822430 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:43.822496 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:43.822717 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:43.822721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:43.822724 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:43.822806 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.822808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.824288 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.824289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.825681 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.825683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.826545 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.826546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.826631 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.826647 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 03:25:43.826771 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 03:25:43.826860 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":42,"thread_start_us":72,"threads_started":1}
I20260927 03:25:43.826967 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.822320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51122 (local address 127.0.98.3:37929)
0927 03:25:43.822537 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:43.822542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:43.822554 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:25:43.822593 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:43.822596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:43.822612 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:43.822702 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:43.823022 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.824167 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.825768 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.826473 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.826526 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.826569 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:43.826790 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 03:25:43.826882 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
I20260927 03:25:43.827378 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.822425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42967 (local address 127.0.0.1:59672)
0927 03:25:43.822813 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:43.824259 (+ 1446us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:43.824311 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:43.824463 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:43.824464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:43.824466 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:43.824509 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.824511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.825355 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.825356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.826271 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.826273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.827166 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.827167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.827226 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.827229 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:25:43.827274 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:25:43.827314 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":78,"threads_started":1}
I20260927 03:25:43.827417 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.822651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.0.98.1:42967)
0927 03:25:43.822959 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:43.822963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:43.824333 (+ 1370us) server_negotiation.cc:407] Connection header received
0927 03:25:43.824379 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:43.824381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:43.824396 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:43.824460 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:43.824566 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.825260 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.826348 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.827012 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.827075 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.827134 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:43.827303 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 03:25:43.827343 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":119,"threads_started":1}
I20260927 03:25:43.827664 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.822223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45867 (local address 127.0.0.1:36578)
0927 03:25:43.822632 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:43.822643 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:43.822692 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:43.822908 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:43.822909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:43.822912 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:43.822954 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.822956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.824253 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.824255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.825681 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.825683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.827438 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.827439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.827512 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.827516 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:25:43.827557 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 03:25:43.827598 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":66,"thread_start_us":56,"threads_started":1}
I20260927 03:25:43.827898 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.822354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.0.98.2:45867)
0927 03:25:43.822554 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:43.822559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:43.822749 (+ 190us) server_negotiation.cc:407] Connection header received
0927 03:25:43.822796 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:43.822798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:43.822813 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:43.822909 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:43.823015 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.824138 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.826602 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.827267 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.827345 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.827404 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:43.827771 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 03:25:43.827820 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
I20260927 03:25:43.828055 15869 tablet_service.cc:1467] Processing CreateTablet for tablet 94007a03bb384e3f862a78cc27ca48d3 (DEFAULT_TABLE table=test-table [id=1dfa55d235334b7388ef8d460c3b41c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:25:43.828473 15869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94007a03bb384e3f862a78cc27ca48d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:43.828531 16003 tablet_service.cc:1467] Processing CreateTablet for tablet 94007a03bb384e3f862a78cc27ca48d3 (DEFAULT_TABLE table=test-table [id=1dfa55d235334b7388ef8d460c3b41c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:25:43.828838 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94007a03bb384e3f862a78cc27ca48d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:43.831283 16217 tablet_bootstrap.cc:492] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: Bootstrap starting.
I20260927 03:25:43.831458 16218 tablet_bootstrap.cc:492] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31: Bootstrap starting.
I20260927 03:25:43.831490 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 94007a03bb384e3f862a78cc27ca48d3 (DEFAULT_TABLE table=test-table [id=1dfa55d235334b7388ef8d460c3b41c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:25:43.831722 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94007a03bb384e3f862a78cc27ca48d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:25:43.832063 16218 tablet_bootstrap.cc:654] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:43.832063 16217 tablet_bootstrap.cc:654] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:43.832628 16217 log.cc:824] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:43.832628 16218 log.cc:824] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:43.833415 16218 tablet_bootstrap.cc:492] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31: No bootstrap required, opened a new log
I20260927 03:25:43.833472 16218 ts_tablet_manager.cc:1397] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:25:43.833563 16217 tablet_bootstrap.cc:492] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: No bootstrap required, opened a new log
I20260927 03:25:43.833609 16217 ts_tablet_manager.cc:1397] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:25:43.834038 16221 tablet_bootstrap.cc:492] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88: Bootstrap starting.
I20260927 03:25:43.834671 16221 tablet_bootstrap.cc:654] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88: Neither blocks nor log segments found. Creating new log.
I20260927 03:25:43.835186 16221 log.cc:824] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88: Log is configured to *not* fsync() on all Append() calls
I20260927 03:25:43.835547 16218 raft_consensus.cc:348] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:25:43.835603 16217 raft_consensus.cc:348] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:25:43.835834 16217 raft_consensus.cc:374] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:43.835821 16218 raft_consensus.cc:374] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:43.835878 16217 raft_consensus.cc:729] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 344fa21d06a94b5a9ed279793f932e22, State: Initialized, Role: FOLLOWER
I20260927 03:25:43.835878 16218 raft_consensus.cc:729] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b09b2401454b29b307204e88c68c31, State: Initialized, Role: FOLLOWER
I20260927 03:25:43.835979 16221 tablet_bootstrap.cc:492] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88: No bootstrap required, opened a new log
I20260927 03:25:43.836047 16221 ts_tablet_manager.cc:1397] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:25:43.836019 16217 consensus_queue.cc:260] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:25:43.836019 16218 consensus_queue.cc:260] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:25:43.836323 16069 heartbeater.cc:503] Master 127.0.98.62:34447 was elected leader, sending a full tablet report...
I20260927 03:25:43.836354 16217 ts_tablet_manager.cc:1428] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:25:43.836479 15935 heartbeater.cc:503] Master 127.0.98.62:34447 was elected leader, sending a full tablet report...
I20260927 03:25:43.836315 16218 ts_tablet_manager.cc:1428] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:25:43.837949 16221 raft_consensus.cc:348] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:25:43.838088 16221 raft_consensus.cc:374] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:25:43.838120 16221 raft_consensus.cc:729] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62e14b49b72140008596d3632165bd88, State: Initialized, Role: FOLLOWER
I20260927 03:25:43.838241 16221 consensus_queue.cc:260] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:25:43.838490 16221 ts_tablet_manager.cc:1428] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 03:25:43.838558 16203 heartbeater.cc:503] Master 127.0.98.62:34447 was elected leader, sending a full tablet report...
I20260927 03:25:43.866272 16223 raft_consensus.cc:482] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:25:43.866395 16223 raft_consensus.cc:504] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:25:43.866694 16223 leader_election.cc:288] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62e14b49b72140008596d3632165bd88 (127.0.98.3:37929), 344fa21d06a94b5a9ed279793f932e22 (127.0.98.1:42967)
I20260927 03:25:43.869402 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.866886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37929 (local address 127.0.98.2:41145)
0927 03:25:43.866936 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:43.866948 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:43.866992 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:43.867143 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:43.867144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:43.867147 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:43.867219 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.867221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.867839 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.867840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.868611 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.868613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.869147 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.869149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.869229 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.869232 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:25:43.869277 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:25:43.869325 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 03:25:43.869402 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.866936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41145 (local address 127.0.98.3:37929)
0927 03:25:43.866985 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:43.866987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:43.866997 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:25:43.867053 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:43.867054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:43.867067 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:43.867122 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:43.867276 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.867744 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.868699 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.869080 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.869140 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.869151 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:43.869287 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 03:25:43.869324 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:25:43.869517 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.867018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42967 (local address 127.0.98.2:39747)
0927 03:25:43.867178 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:43.867186 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:25:43.867234 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:25:43.867359 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:25:43.867360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:25:43.867361 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:25:43.867430 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.867432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.868007 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.868007 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.868740 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:25:43.868742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.869278 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.869279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:25:43.869352 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.869356 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:25:43.869399 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:25:43.869435 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260927 03:25:43.869509 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.867121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39747 (local address 127.0.98.1:42967)
0927 03:25:43.867165 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:43.867167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:43.867236 (+ 69us) server_negotiation.cc:407] Connection header received
0927 03:25:43.867291 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:43.867292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:43.867304 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:43.867356 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:25:43.867482 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.867939 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.868836 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.869213 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.869269 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.869286 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:43.869408 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 03:25:43.869447 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:25:43.869761 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94007a03bb384e3f862a78cc27ca48d3" candidate_uuid: "66b09b2401454b29b307204e88c68c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62e14b49b72140008596d3632165bd88" is_pre_election: true
I20260927 03:25:43.869807 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94007a03bb384e3f862a78cc27ca48d3" candidate_uuid: "66b09b2401454b29b307204e88c68c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "344fa21d06a94b5a9ed279793f932e22" is_pre_election: true
I20260927 03:25:43.869940 16157 raft_consensus.cc:2454] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b09b2401454b29b307204e88c68c31 in term 0.
I20260927 03:25:43.869940 15889 raft_consensus.cc:2454] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b09b2401454b29b307204e88c68c31 in term 0.
I20260927 03:25:43.870157 15959 leader_election.cc:302] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62e14b49b72140008596d3632165bd88, 66b09b2401454b29b307204e88c68c31; no voters:
I20260927 03:25:43.870358 16223 raft_consensus.cc:2781] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:25:43.870425 16223 raft_consensus.cc:482] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:25:43.870441 16223 raft_consensus.cc:3037] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:43.871222 16223 raft_consensus.cc:504] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:25:43.871352 16223 leader_election.cc:288] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [CANDIDATE]: Term 1 election: Requested vote from peers 62e14b49b72140008596d3632165bd88 (127.0.98.3:37929), 344fa21d06a94b5a9ed279793f932e22 (127.0.98.1:42967)
I20260927 03:25:43.871532 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94007a03bb384e3f862a78cc27ca48d3" candidate_uuid: "66b09b2401454b29b307204e88c68c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "344fa21d06a94b5a9ed279793f932e22"
I20260927 03:25:43.871526 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94007a03bb384e3f862a78cc27ca48d3" candidate_uuid: "66b09b2401454b29b307204e88c68c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62e14b49b72140008596d3632165bd88"
I20260927 03:25:43.871610 16157 raft_consensus.cc:3037] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:43.871610 15889 raft_consensus.cc:3037] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:25:43.872470 15889 raft_consensus.cc:2454] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b09b2401454b29b307204e88c68c31 in term 1.
I20260927 03:25:43.872474 16157 raft_consensus.cc:2454] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b09b2401454b29b307204e88c68c31 in term 1.
I20260927 03:25:43.872658 15959 leader_election.cc:302] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62e14b49b72140008596d3632165bd88, 66b09b2401454b29b307204e88c68c31; no voters:
I20260927 03:25:43.872745 16223 raft_consensus.cc:2781] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:25:43.872885 16223 raft_consensus.cc:686] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [term 1 LEADER]: Becoming Leader. State: Replica: 66b09b2401454b29b307204e88c68c31, State: Running, Role: LEADER
I20260927 03:25:43.872972 16223 consensus_queue.cc:237] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:25:43.873848 15743 catalog_manager.cc:5495] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66b09b2401454b29b307204e88c68c31 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "66b09b2401454b29b307204e88c68c31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } health_report { overall_health: UNKNOWN } } }
I20260927 03:25:43.890965 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.888480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36590 (local address 127.0.98.2:45867)
0927 03:25:43.888538 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:25:43.888539 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:25:43.888547 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:25:43.888582 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:25:43.888583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:25:43.888625 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:25:43.888692 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:25:43.888883 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.889434 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.890264 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:25:43.890455 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:25:43.890511 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:25:43.890638 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:25:43.890745 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:25:43.890811 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:25:43.890867 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 03:25:43.890891 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260927 03:25:43.892464 16070 tablet.cc:2369] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:25:43.892887 16157 raft_consensus.cc:1261] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88 [term 1 FOLLOWER]: Refusing update from remote peer 66b09b2401454b29b307204e88c68c31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:25:43.892894 15889 raft_consensus.cc:1261] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Refusing update from remote peer 66b09b2401454b29b307204e88c68c31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:25:43.893263 16227 consensus_queue.cc:1035] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:25:43.893424 16223 consensus_queue.cc:1035] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:25:43.895948 16231 mvcc.cc:205] Tried to move back new op lower bound from 7333804211782443008 to 7333804211704504320. Current Snapshot: MvccSnapshot[applied={T|T < 7333804211782443008}]
I20260927 03:25:43.896255 16230 mvcc.cc:205] Tried to move back new op lower bound from 7333804211782443008 to 7333804211704504320. Current Snapshot: MvccSnapshot[applied={T|T < 7333804211782443008}]
I20260927 03:25:43.898326 15742 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:25:43.898414 15742 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:25:43.899165 15742 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Sending DeleteTablet for 3 replicas of tablet 94007a03bb384e3f862a78cc27ca48d3
I20260927 03:25:43.899518 15869 tablet_service.cc:1514] Processing DeleteTablet for tablet 94007a03bb384e3f862a78cc27ca48d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:25:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59672
I20260927 03:25:43.899513 16137 tablet_service.cc:1514] Processing DeleteTablet for tablet 94007a03bb384e3f862a78cc27ca48d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:25:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51122
I20260927 03:25:43.899734 16241 tablet_replica.cc:330] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: stopping tablet replica
I20260927 03:25:43.899735 16003 tablet_service.cc:1514] Processing DeleteTablet for tablet 94007a03bb384e3f862a78cc27ca48d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:25:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36578
I20260927 03:25:43.899852 16241 raft_consensus.cc:2229] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:25:43.899940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 15806
I20260927 03:25:43.899966 16241 raft_consensus.cc:2258] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:25:43.899967 16242 tablet_replica.cc:330] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31: stopping tablet replica
I20260927 03:25:43.900161 16242 raft_consensus.cc:2229] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:25:43.900318 16242 raft_consensus.cc:2258] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:25:43.900887 16242 ts_tablet_manager.cc:1905] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:25:43.902906 16242 ts_tablet_manager.cc:1918] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:25:43.902984 16242 log.cc:1196] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/94007a03bb384e3f862a78cc27ca48d3
I20260927 03:25:43.903249 16242 ts_tablet_manager.cc:1939] T 94007a03bb384e3f862a78cc27ca48d3 P 66b09b2401454b29b307204e88c68c31: Deleting consensus metadata
I20260927 03:25:43.903661 15730 catalog_manager.cc:4841] TS 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867): tablet 94007a03bb384e3f862a78cc27ca48d3 (table test-table [id=1dfa55d235334b7388ef8d460c3b41c9]) successfully deleted
W20260927 03:25:43.905212 15731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42967 (error 108)
I20260927 03:25:43.905723 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:25:43.905608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42967 (local address 127.0.0.1:59682)
0927 03:25:43.905656 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:25:43.905688 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:25:43.905839 15731 catalog_manager.cc:4584] TS 344fa21d06a94b5a9ed279793f932e22 (127.0.98.1:42967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94007a03bb384e3f862a78cc27ca48d3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42967: connect: Connection refused (error 111)
I20260927 03:25:43.906160 16240 tablet_replica.cc:330] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88: stopping tablet replica
I20260927 03:25:43.906298 16240 raft_consensus.cc:2229] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:25:43.906423 16240 raft_consensus.cc:2258] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:25:43.906643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 15940
I20260927 03:25:43.906911 16240 ts_tablet_manager.cc:1905] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:25:43.909026 16240 ts_tablet_manager.cc:1918] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:25:43.909096 16240 log.cc:1196] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/94007a03bb384e3f862a78cc27ca48d3
I20260927 03:25:43.909346 16240 ts_tablet_manager.cc:1939] T 94007a03bb384e3f862a78cc27ca48d3 P 62e14b49b72140008596d3632165bd88: Deleting consensus metadata
I20260927 03:25:43.909734 15731 catalog_manager.cc:4841] TS 62e14b49b72140008596d3632165bd88 (127.0.98.3:37929): tablet 94007a03bb384e3f862a78cc27ca48d3 (table test-table [id=1dfa55d235334b7388ef8d460c3b41c9]) successfully deleted
I20260927 03:25:43.911566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 16074
I20260927 03:25:43.916369 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 15712
I20260927 03:26:03.921732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34447
--webserver_interface=127.0.98.62
--webserver_port=42767
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:04.001582 16244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:04.001791 16244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:04.001812 16244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:04.002969 16244 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 03:26:04.003005 16244 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 03:26:04.003017 16244 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 03:26:04.003026 16244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:04.003038 16244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:04.003049 16244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:04.003060 16244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:04.003072 16244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:04.004515 16244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:04.004714 16244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:04.004904 16244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:04.013504 16251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:04.013522 16253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:04.013538 16244 server_base.cc:1034] running on GCE node
W20260927 03:26:04.013517 16250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:04.014060 16244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:04.014372 16244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:04.015530 16244 hybrid_clock.cc:648] HybridClock initialized: now 1790479564015517 us; error 29 us; skew 500 ppm
Sep 27 03:26:04 dist-test-slave-gw2v krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:04.029992 16244 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:26:04.030510 16244 webserver.cc:460] Webserver started at http://127.0.98.62:42767/ using document root <none> and password file <none>
I20260927 03:26:04.030753 16244 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:04.030836 16244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:04.032398 16244 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:04.033277 16260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:04.033494 16244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:04.033599 16244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "c8c0a65633f045f499723665fdc34fbc"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "c14cf596fbaf9c94d82822fba04e386c"
server_key_iv: "f3b1c923280cef4ed6699a0f10e7bc98"
server_key_version: "encryptionkey@0"
I20260927 03:26:04.034055 16244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:04.050614 16244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:04.052891 16244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:04.053040 16244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:04.057168 16244 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34447
I20260927 03:26:04.057217 16312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34447 every 8 connection(s)
I20260927 03:26:04.057578 16244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:04.059077 16313 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:26:04.059687 16313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Bootstrap starting.
I20260927 03:26:04.067173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 16244
I20260927 03:26:04.067525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42967
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34447
--webserver_port=38795
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:26:04.067672 16313 log.cc:824] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:04.070283 16313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:26:04.070592 16313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Bootstrap complete.
I20260927 03:26:04.073944 16313 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } }
I20260927 03:26:04.074307 16313 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c0a65633f045f499723665fdc34fbc, State: Initialized, Role: FOLLOWER
I20260927 03:26:04.074517 16313 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } }
I20260927 03:26:04.074607 16313 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:04.074662 16313 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:04.074713 16313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:26:04.076246 16313 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } }
I20260927 03:26:04.076368 16313 leader_election.cc:302] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8c0a65633f045f499723665fdc34fbc; no voters:
I20260927 03:26:04.076565 16313 leader_election.cc:288] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:26:04.076622 16317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:26:04.076790 16317 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 2 LEADER]: Becoming Leader. State: Replica: c8c0a65633f045f499723665fdc34fbc, State: Running, Role: LEADER
I20260927 03:26:04.076846 16313 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:04.076893 16317 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } }
I20260927 03:26:04.077160 16317 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8c0a65633f045f499723665fdc34fbc. Latest consensus state: current_term: 2 leader_uuid: "c8c0a65633f045f499723665fdc34fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } } }
I20260927 03:26:04.077175 16318 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c8c0a65633f045f499723665fdc34fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } } }
I20260927 03:26:04.077241 16317 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:04.077246 16318 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:04.077533 16323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:04.078328 16323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:04.078445 16323 catalog_manager.cc:1259] Loaded cluster ID: a6923a81258848978b234a924ee2b699
I20260927 03:26:04.078480 16323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:26:04.079989 16323 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:04.080180 16323 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Loaded TSK: 0
I20260927 03:26:04.098752 16323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Generated new TSK 1
I20260927 03:26:04.098891 16323 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:26:04.151298 16316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:04.151527 16316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:04.151561 16316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:04.152814 16316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:04.152860 16316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:04.152908 16316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:26:04.154443 16316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38795
--tserver_master_addrs=127.0.98.62:34447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:04.154704 16316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:04.154974 16316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:04.160013 16341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:04.160024 16340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:04.160122 16316 server_base.cc:1034] running on GCE node
W20260927 03:26:04.160022 16343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:04.160490 16316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:04.160739 16316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:04.161898 16316 hybrid_clock.cc:648] HybridClock initialized: now 1790479564161881 us; error 31 us; skew 500 ppm
Sep 27 03:26:04 dist-test-slave-gw2v krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:04.164772 16316 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:26:04.165231 16316 webserver.cc:460] Webserver started at http://127.0.98.1:38795/ using document root <none> and password file <none>
I20260927 03:26:04.165463 16316 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:04.165517 16316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:04.166893 16316 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:04.167568 16350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:04.167752 16316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:04.167831 16316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "344fa21d06a94b5a9ed279793f932e22"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "530e3e60fce0c5342da60805f7f09bdb"
server_key_iv: "5c6c801959963757230cd9c84465a6bb"
server_key_version: "encryptionkey@0"
I20260927 03:26:04.168186 16316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:04.184684 16316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:04.186867 16316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:04.187011 16316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:04.187237 16316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:04.187711 16357 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:26:04.189105 16316 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:26:04.189155 16316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:26:04.189180 16316 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:26:04.189774 16316 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:26:04.189814 16316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:26:04.189896 16357 tablet_bootstrap.cc:492] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: Bootstrap starting.
I20260927 03:26:04.197273 16316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42967
I20260927 03:26:04.197438 16464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42967 every 8 connection(s)
I20260927 03:26:04.197822 16316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:26:04.200667 16357 log.cc:824] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:04.203084 16357 tablet_bootstrap.cc:492] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:26:04.203255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 16316
I20260927 03:26:04.203364 16357 tablet_bootstrap.cc:492] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: Bootstrap complete.
I20260927 03:26:04.203513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45867
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34447
--webserver_port=42145
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:26:04.204375 16357 ts_tablet_manager.cc:1397] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260927 03:26:04.205567 16357 raft_consensus.cc:348] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:04.205797 16357 raft_consensus.cc:729] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 344fa21d06a94b5a9ed279793f932e22, State: Initialized, Role: FOLLOWER
I20260927 03:26:04.205932 16357 consensus_queue.cc:260] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:04.206158 16357 ts_tablet_manager.cc:1428] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
Sep 27 03:26:04 dist-test-slave-gw2v krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:26:04.210675 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.198502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:37935)
0927 03:26:04.198763 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:04.198804 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:04.199420 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:04.199960 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:04.199964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:04.200309 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:04.200572 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.200578 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.202260 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.202261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.203541 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.203545 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.204726 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.204728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.204874 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.205488 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:26:04.205508 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:04.207270 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:04.208944 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:04.208950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:04.208952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:04.209217 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:04.209505 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:04.209505 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:04.209506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:04.209629 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:04.210100 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:04.210106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:04.210242 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 03:26:04.210319 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:26:04.210474 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":82,"threads_started":1}
I20260927 03:26:04.211390 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.198638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37935 (local address 127.0.98.62:34447)
0927 03:26:04.198931 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:04.198942 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:04.198993 (+ 51us) server_negotiation.cc:407] Connection header received
0927 03:26:04.199664 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:04.199671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:04.199742 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:04.199944 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:04.200870 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.202126 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.203895 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.204599 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.205177 (+ 578us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.207370 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:04.207373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:04.207373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:04.207383 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:04.208867 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:04.209308 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:04.209312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:04.209312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:04.209438 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:04.209707 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:04.209708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:04.209708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:04.210028 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:04.210950 (+ 922us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:04.211024 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 03:26:04.211201 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":95,"threads_started":1}
I20260927 03:26:04.212327 16465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34447
I20260927 03:26:04.212414 16465 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:04.212604 16465 heartbeater.cc:511] Master 127.0.98.62:34447 requested a full tablet report, sending...
I20260927 03:26:04.213340 16277 ts_manager.cc:194] Registered new tserver with Master: 344fa21d06a94b5a9ed279793f932e22 (127.0.98.1:42967)
I20260927 03:26:04.214426 16277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37935
I20260927 03:26:04.215476 16465 heartbeater.cc:503] Master 127.0.98.62:34447 was elected leader, sending a full tablet report...
I20260927 03:26:04.219286 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.214689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.0.98.1:42967)
0927 03:26:04.214879 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:04.214884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:04.215948 (+ 1064us) server_negotiation.cc:407] Connection header received
0927 03:26:04.216097 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:04.216100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:04.216114 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:04.216196 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:04.216347 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.216961 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.218424 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.218954 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.219015 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.219051 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:04.219212 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 03:26:04.219241 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
I20260927 03:26:04.219328 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.213970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42967 (local address 127.0.0.1:33952)
0927 03:26:04.215836 (+ 1866us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:04.215885 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:04.215989 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:04.216208 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:04.216211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:04.216214 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:04.216280 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.216281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.217025 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.217026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.218331 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.218334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.219021 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.219022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.219118 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.219132 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:26:04.219193 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:26:04.219243 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1835,"thread_start_us":84,"threads_started":1}
I20260927 03:26:04.219660 16399 tablet_service.cc:1514] Processing DeleteTablet for tablet 94007a03bb384e3f862a78cc27ca48d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:25:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33952
I20260927 03:26:04.220150 16474 tablet_replica.cc:330] stopping tablet replica
I20260927 03:26:04.220234 16474 raft_consensus.cc:2229] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:26:04.220296 16474 raft_consensus.cc:2258] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:04.220613 16474 ts_tablet_manager.cc:1905] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:26:04.222718 16474 ts_tablet_manager.cc:1918] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:26:04.222784 16474 log.cc:1196] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal/wals/94007a03bb384e3f862a78cc27ca48d3
I20260927 03:26:04.223006 16474 ts_tablet_manager.cc:1939] T 94007a03bb384e3f862a78cc27ca48d3 P 344fa21d06a94b5a9ed279793f932e22: Deleting consensus metadata
I20260927 03:26:04.223426 16264 catalog_manager.cc:4841] TS 344fa21d06a94b5a9ed279793f932e22 (127.0.98.1:42967): tablet 94007a03bb384e3f862a78cc27ca48d3 (table test-table [id=1dfa55d235334b7388ef8d460c3b41c9]) successfully deleted
W20260927 03:26:04.290158 16470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:04.290385 16470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:04.290407 16470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:04.291579 16470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:04.291620 16470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:04.291644 16470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:26:04.293061 16470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42145
--tserver_master_addrs=127.0.98.62:34447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:04.293256 16470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:04.293444 16470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:04.298240 16482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:04.298416 16470 server_base.cc:1034] running on GCE node
W20260927 03:26:04.298280 16480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:04.298237 16479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:04.298719 16470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:04.298990 16470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:04.300137 16470 hybrid_clock.cc:648] HybridClock initialized: now 1790479564300122 us; error 28 us; skew 500 ppm
Sep 27 03:26:04 dist-test-slave-gw2v krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:04.302608 16470 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:26:04.303117 16470 webserver.cc:460] Webserver started at http://127.0.98.2:42145/ using document root <none> and password file <none>
I20260927 03:26:04.303356 16470 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:04.303423 16470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:04.304819 16470 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:04.305440 16489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:04.305612 16470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:26:04.305693 16470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "66b09b2401454b29b307204e88c68c31"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "469924b985dc159629fc406182f0331e"
server_key_iv: "06a1f6a539187217a7551ae829475bb7"
server_key_version: "encryptionkey@0"
I20260927 03:26:04.306046 16470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:04.325726 16470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:04.327637 16470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:04.327773 16470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:04.327994 16470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:04.328281 16470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:04.328315 16470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:04.328344 16470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:04.328366 16470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:04.333768 16470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45867
I20260927 03:26:04.333810 16602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45867 every 8 connection(s)
I20260927 03:26:04.334146 16470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:26:04 dist-test-slave-gw2v krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:26:04.342087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 16470
I20260927 03:26:04.342519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37929
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34447
--webserver_port=44131
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:26:04.343047 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.334880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51835 (local address 127.0.98.62:34447)
0927 03:26:04.334934 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:04.334936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:04.335122 (+ 186us) server_negotiation.cc:407] Connection header received
0927 03:26:04.335586 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:04.335587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:04.335596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:04.335650 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:04.336594 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.337113 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.338192 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.338403 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.338452 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.340368 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:04.340369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:04.340370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:04.340376 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:04.341465 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:04.341871 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:04.341872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:04.341872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:04.341917 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:04.342324 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:04.342325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:04.342325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:04.342454 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:04.342504 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:04.342906 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 03:26:04.342977 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:26:04.343281 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.334856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:51835)
0927 03:26:04.335000 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:04.335041 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:04.335512 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:04.335705 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:04.335709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:04.336254 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:04.336500 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.336505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.337208 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.337209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.338113 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.338116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.338459 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.338459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.338548 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.338973 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:26:04.338981 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:04.340256 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:04.341556 (+ 1300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:04.341562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:04.341564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:04.341789 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:04.341986 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:04.341986 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:04.341987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:04.342253 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:04.342539 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:04.342547 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:04.342710 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 03:26:04.342887 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 03:26:04.343059 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260927 03:26:04.344245 16603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34447
I20260927 03:26:04.344336 16603 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:04.344492 16603 heartbeater.cc:511] Master 127.0.98.62:34447 requested a full tablet report, sending...
I20260927 03:26:04.344816 16277 ts_manager.cc:194] Registered new tserver with Master: 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867)
I20260927 03:26:04.345149 16277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51835
W20260927 03:26:04.422165 16606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:04.422389 16606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:04.422420 16606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:04.423599 16606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:04.423641 16606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:04.423676 16606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:26:04.425102 16606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44131
--tserver_master_addrs=127.0.98.62:34447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:04.425330 16606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:04.425537 16606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:04.430301 16612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:04.430305 16611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:04.430305 16614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:04.430799 16606 server_base.cc:1034] running on GCE node
I20260927 03:26:04.431028 16606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:04.431259 16606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:04.432403 16606 hybrid_clock.cc:648] HybridClock initialized: now 1790479564432391 us; error 27 us; skew 500 ppm
Sep 27 03:26:04 dist-test-slave-gw2v krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:04.434768 16606 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:26:04.435197 16606 webserver.cc:460] Webserver started at http://127.0.98.3:44131/ using document root <none> and password file <none>
I20260927 03:26:04.435398 16606 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:04.435460 16606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:04.436815 16606 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:26:04.437373 16621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:04.437522 16606 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:26:04.437603 16606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "62e14b49b72140008596d3632165bd88"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "d8d89862e5b6c60f2f9ffcb3dbd84b67"
server_key_iv: "7c1b9319d5758c0371ca9d523b819b9f"
server_key_version: "encryptionkey@0"
I20260927 03:26:04.437943 16606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:04.448814 16606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:04.450534 16606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:04.450664 16606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:04.450909 16606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:04.451203 16606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:04.451236 16606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:04.451264 16606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:04.451287 16606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:04.456534 16606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37929
I20260927 03:26:04.456590 16734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37929 every 8 connection(s)
I20260927 03:26:04.456899 16606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:26:04.457440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 16606
Sep 27 03:26:04 dist-test-slave-gw2v krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:26:04.464576 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.457811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.0.98.62:34447)
0927 03:26:04.457919 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:04.457921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:04.458023 (+ 102us) server_negotiation.cc:407] Connection header received
0927 03:26:04.458086 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:04.458087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:04.458099 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:04.458161 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:04.458764 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.459542 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.460590 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.460881 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.460950 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.462197 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:04.462198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:04.462198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:04.462205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:04.463483 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:04.463812 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:04.463813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:04.463814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:04.463901 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:04.464058 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:04.464059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:04.464060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:04.464184 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:04.464311 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:04.464395 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 03:26:04.464474 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
I20260927 03:26:04.466492 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.457535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38167 (local address 127.0.98.62:34447)
0927 03:26:04.457604 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:04.457606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:04.457808 (+ 202us) server_negotiation.cc:407] Connection header received
0927 03:26:04.458423 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:04.458424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:04.458434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:04.458496 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:04.459625 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.460364 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.461658 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.461940 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.462014 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.464132 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:04.464133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:04.464133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:04.464138 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:04.465518 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:04.465837 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:04.465837 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:04.465837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:04.465878 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:04.466036 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:04.466036 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:04.466036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:04.466118 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:04.466157 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:04.466377 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 03:26:04.466430 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:26:04.466675 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.457536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:38167)
0927 03:26:04.457690 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:04.457725 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:04.458330 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:04.458552 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:04.458556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:04.459280 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:04.459528 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.459534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.460448 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.460449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.461579 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.461582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.462016 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.462017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.462107 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.462523 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:26:04.462532 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:04.464061 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:04.465580 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:04.465585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:04.465587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:04.465784 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:04.465913 (+ 129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:04.465913 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:04.465913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:04.465997 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:04.466151 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:04.466154 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:04.466260 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 03:26:04.466364 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 03:26:04.466482 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260927 03:26:04.467111 16735 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34447
I20260927 03:26:04.467206 16735 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:04.467386 16735 heartbeater.cc:511] Master 127.0.98.62:34447 requested a full tablet report, sending...
I20260927 03:26:04.467711 16277 ts_manager.cc:194] Registered new tserver with Master: 62e14b49b72140008596d3632165bd88 (127.0.98.3:37929)
I20260927 03:26:04.468055 16277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38167
I20260927 03:26:04.476152 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:26:04.478802 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.476530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.0.98.62:34447)
0927 03:26:04.476603 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:04.476606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:04.476695 (+ 89us) server_negotiation.cc:407] Connection header received
0927 03:26:04.476742 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:04.476743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:04.476783 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:04.476839 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:26:04.476940 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.477440 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.478235 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.478424 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.478484 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.478584 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:26:04.478629 (+ 45us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 03:26:04.478749 (+ 120us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43294: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 03:26:04.478933 16468 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43294: token expired
W20260927 03:26:04.479240 16208 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:34447)
I20260927 03:26:04.485561 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.479482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.0.98.62:34447)
0927 03:26:04.479521 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:04.479523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:04.479533 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:26:04.479555 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:04.479556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:04.479564 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:04.479624 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:04.480097 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.480777 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.481672 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.481951 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.482016 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.483149 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:04.483150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:04.483151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:04.483156 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:04.484572 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:04.484893 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:04.484894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:04.484895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:04.484949 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:04.485094 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:04.485095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:04.485095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:04.485266 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:04.485365 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:04.485419 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 03:26:04.485486 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:26:04.486691 16277 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:26:04.488255 16277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:26:04.490422 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 486227f69c254f0bbdcbb4fe64619be2 (DEFAULT_TABLE table=test-table [id=fa46a46266284ddfbef903a700e86ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:04.490615 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486227f69c254f0bbdcbb4fe64619be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:04.492064 16357 tablet_bootstrap.cc:492] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22: Bootstrap starting.
I20260927 03:26:04.492355 16357 tablet_bootstrap.cc:654] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:04.492826 16357 tablet_bootstrap.cc:492] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22: No bootstrap required, opened a new log
I20260927 03:26:04.492883 16357 ts_tablet_manager.cc:1397] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:26:04.492987 16357 raft_consensus.cc:348] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:04.493060 16357 raft_consensus.cc:374] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:04.493108 16357 raft_consensus.cc:729] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 344fa21d06a94b5a9ed279793f932e22, State: Initialized, Role: FOLLOWER
I20260927 03:26:04.493155 16357 consensus_queue.cc:260] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:04.493238 16357 ts_tablet_manager.cc:1428] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:04.493477 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.489959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45867 (local address 127.0.0.1:33926)
0927 03:26:04.490007 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:04.490018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:04.490056 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:04.490392 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:04.490393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:04.490395 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:04.490439 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.490440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.491497 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.491498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.492480 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.492482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.493265 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.493266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.493328 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.493332 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:26:04.493377 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:26:04.493415 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 03:26:04.494297 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.490011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37929 (local address 127.0.0.1:48408)
0927 03:26:04.490188 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:04.490200 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:04.490248 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:04.490665 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:04.490666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:04.490668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:04.490703 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.490704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.491826 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.491827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.492836 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.492839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.494123 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.494123 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.494169 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.494172 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:04.494216 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:26:04.494248 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":117,"threads_started":1}
I20260927 03:26:04.494357 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.490335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48408 (local address 127.0.98.3:37929)
0927 03:26:04.490513 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:04.490517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:04.490527 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:26:04.490564 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:04.490566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:04.490580 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:04.490654 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:04.490754 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.491734 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.493411 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.494069 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.494119 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.494158 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:04.494243 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:26:04.494276 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
I20260927 03:26:04.494349 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.490050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33926 (local address 127.0.98.2:45867)
0927 03:26:04.490221 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:04.490226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:04.490237 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:26:04.490276 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:04.490278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:04.490294 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:04.490504 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:04.490593 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.491425 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.492553 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.493163 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.493228 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.494190 (+ 962us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:04.494214 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 03:26:04.494259 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260927 03:26:04.495039 16669 tablet_service.cc:1467] Processing CreateTablet for tablet 486227f69c254f0bbdcbb4fe64619be2 (DEFAULT_TABLE table=test-table [id=fa46a46266284ddfbef903a700e86ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:04.495039 16537 tablet_service.cc:1467] Processing CreateTablet for tablet 486227f69c254f0bbdcbb4fe64619be2 (DEFAULT_TABLE table=test-table [id=fa46a46266284ddfbef903a700e86ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:04.495360 16669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486227f69c254f0bbdcbb4fe64619be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:04.495360 16537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486227f69c254f0bbdcbb4fe64619be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:04.498706 16745 tablet_bootstrap.cc:492] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88: Bootstrap starting.
I20260927 03:26:04.498955 16746 tablet_bootstrap.cc:492] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31: Bootstrap starting.
I20260927 03:26:04.499361 16745 tablet_bootstrap.cc:654] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:04.499475 16746 tablet_bootstrap.cc:654] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:04.499833 16745 log.cc:824] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:04.499933 16746 log.cc:824] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:04.500563 16746 tablet_bootstrap.cc:492] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31: No bootstrap required, opened a new log
I20260927 03:26:04.500586 16745 tablet_bootstrap.cc:492] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88: No bootstrap required, opened a new log
I20260927 03:26:04.500627 16745 ts_tablet_manager.cc:1397] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:26:04.500620 16746 ts_tablet_manager.cc:1397] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:26:04.501971 16745 raft_consensus.cc:348] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:04.501991 16746 raft_consensus.cc:348] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:04.502108 16745 raft_consensus.cc:374] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:04.502112 16746 raft_consensus.cc:374] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:04.502130 16745 raft_consensus.cc:729] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62e14b49b72140008596d3632165bd88, State: Initialized, Role: FOLLOWER
I20260927 03:26:04.502141 16746 raft_consensus.cc:729] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b09b2401454b29b307204e88c68c31, State: Initialized, Role: FOLLOWER
I20260927 03:26:04.502230 16745 consensus_queue.cc:260] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:04.502278 16746 consensus_queue.cc:260] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:04.502512 16735 heartbeater.cc:503] Master 127.0.98.62:34447 was elected leader, sending a full tablet report...
I20260927 03:26:04.502542 16746 ts_tablet_manager.cc:1428] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:26:04.502578 16603 heartbeater.cc:503] Master 127.0.98.62:34447 was elected leader, sending a full tablet report...
I20260927 03:26:04.502501 16745 ts_tablet_manager.cc:1428] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260927 03:26:04.584679 16604 tablet.cc:2369] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:26:04.664121 16749 raft_consensus.cc:482] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:26:04.664253 16749 raft_consensus.cc:504] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:04.664644 16749 leader_election.cc:288] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867), 344fa21d06a94b5a9ed279793f932e22 (127.0.98.1:42967)
I20260927 03:26:04.667668 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.664946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60161 (local address 127.0.98.1:42967)
0927 03:26:04.665010 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:04.665012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:04.665129 (+ 117us) server_negotiation.cc:407] Connection header received
0927 03:26:04.665167 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:04.665168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:04.665184 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:04.665259 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:04.665403 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.665879 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.666939 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.667328 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.667392 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.667402 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:04.667552 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 03:26:04.667604 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:26:04.667676 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.664870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42967 (local address 127.0.98.3:60161)
0927 03:26:04.665061 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:04.665069 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:04.665116 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:04.665261 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:04.665262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:04.665263 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:04.665331 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.665333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.665961 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.665962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.666787 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.666790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.667415 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.667416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.667486 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.667491 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:26:04.667541 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:26:04.667585 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":32,"thread_start_us":107,"threads_started":1}
I20260927 03:26:04.667773 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.664960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56335 (local address 127.0.98.2:45867)
0927 03:26:04.664999 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:04.665000 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:04.665008 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:26:04.665036 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:04.665037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:04.665049 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:04.665105 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:04.665376 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.665867 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.666984 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.667386 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.667486 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.667497 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:04.667655 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 03:26:04.667704 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:26:04.667984 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.664835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45867 (local address 127.0.98.3:56335)
0927 03:26:04.664899 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:04.664912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:04.664960 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:04.665194 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:04.665196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:04.665198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:04.665281 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.665283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.665952 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.665953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.666794 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:04.666797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.667496 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.667497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:04.667583 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.667586 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:04.667636 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:26:04.667680 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":37}
I20260927 03:26:04.668062 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486227f69c254f0bbdcbb4fe64619be2" candidate_uuid: "62e14b49b72140008596d3632165bd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "344fa21d06a94b5a9ed279793f932e22" is_pre_election: true
I20260927 03:26:04.668272 16419 raft_consensus.cc:2454] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62e14b49b72140008596d3632165bd88 in term 0.
I20260927 03:26:04.668409 16557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486227f69c254f0bbdcbb4fe64619be2" candidate_uuid: "62e14b49b72140008596d3632165bd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66b09b2401454b29b307204e88c68c31" is_pre_election: true
I20260927 03:26:04.668535 16625 leader_election.cc:302] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 344fa21d06a94b5a9ed279793f932e22, 62e14b49b72140008596d3632165bd88; no voters:
I20260927 03:26:04.668573 16557 raft_consensus.cc:2454] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62e14b49b72140008596d3632165bd88 in term 0.
I20260927 03:26:04.668682 16749 raft_consensus.cc:2781] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:26:04.668730 16749 raft_consensus.cc:482] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:26:04.668751 16749 raft_consensus.cc:3037] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:04.669466 16749 raft_consensus.cc:504] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:04.669600 16749 leader_election.cc:288] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [CANDIDATE]: Term 1 election: Requested vote from peers 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867), 344fa21d06a94b5a9ed279793f932e22 (127.0.98.1:42967)
I20260927 03:26:04.669767 16557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486227f69c254f0bbdcbb4fe64619be2" candidate_uuid: "62e14b49b72140008596d3632165bd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66b09b2401454b29b307204e88c68c31"
I20260927 03:26:04.669766 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486227f69c254f0bbdcbb4fe64619be2" candidate_uuid: "62e14b49b72140008596d3632165bd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "344fa21d06a94b5a9ed279793f932e22"
I20260927 03:26:04.669852 16419 raft_consensus.cc:3037] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:04.669859 16557 raft_consensus.cc:3037] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:04.670795 16557 raft_consensus.cc:2454] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62e14b49b72140008596d3632165bd88 in term 1.
I20260927 03:26:04.670997 16624 leader_election.cc:302] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62e14b49b72140008596d3632165bd88, 66b09b2401454b29b307204e88c68c31; no voters:
I20260927 03:26:04.671105 16749 raft_consensus.cc:2781] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:04.671147 16419 raft_consensus.cc:2454] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62e14b49b72140008596d3632165bd88 in term 1.
I20260927 03:26:04.671276 16749 raft_consensus.cc:686] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [term 1 LEADER]: Becoming Leader. State: Replica: 62e14b49b72140008596d3632165bd88, State: Running, Role: LEADER
I20260927 03:26:04.671378 16749 consensus_queue.cc:237] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:04.672178 16277 catalog_manager.cc:5495] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62e14b49b72140008596d3632165bd88 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "62e14b49b72140008596d3632165bd88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } health_report { overall_health: UNKNOWN } } }
W20260927 03:26:04.698729 16466 tablet.cc:2369] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:26:04.707407 16736 tablet.cc:2369] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:26:04.720981 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.718746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.0.98.3:37929)
0927 03:26:04.718802 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:04.718804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:04.718811 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:26:04.718875 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:04.718876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:04.718900 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:04.718963 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:26:04.719079 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.719548 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.720271 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:04.720466 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:04.720518 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:04.720669 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:26:04.720771 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:26:04.720843 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:04.720862 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 03:26:04.720884 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:26:04.722770 16419 raft_consensus.cc:1261] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Refusing update from remote peer 62e14b49b72140008596d3632165bd88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:26:04.722770 16557 raft_consensus.cc:1261] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31 [term 1 FOLLOWER]: Refusing update from remote peer 62e14b49b72140008596d3632165bd88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:26:04.723146 16749 consensus_queue.cc:1035] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:26:04.723352 16752 consensus_queue.cc:1035] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:26:04.725238 16756 mvcc.cc:205] Tried to move back new op lower bound from 7333804297101746176 to 7333804296894828544. Current Snapshot: MvccSnapshot[applied={T|T < 7333804297101746176}]
I20260927 03:26:04.727433 16276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:26:04.727535 16276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:26:04.728308 16276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Sending DeleteTablet for 3 replicas of tablet 486227f69c254f0bbdcbb4fe64619be2
I20260927 03:26:04.728690 16669 tablet_service.cc:1514] Processing DeleteTablet for tablet 486227f69c254f0bbdcbb4fe64619be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48408
I20260927 03:26:04.728755 16399 tablet_service.cc:1514] Processing DeleteTablet for tablet 486227f69c254f0bbdcbb4fe64619be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33952
I20260927 03:26:04.728947 16764 tablet_replica.cc:330] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88: stopping tablet replica
I20260927 03:26:04.728966 16537 tablet_service.cc:1514] Processing DeleteTablet for tablet 486227f69c254f0bbdcbb4fe64619be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33926
I20260927 03:26:04.729050 16764 raft_consensus.cc:2229] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:26:04.729152 16764 raft_consensus.cc:2258] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:04.729241 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 16316
I20260927 03:26:04.730090 16766 tablet_replica.cc:330] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31: stopping tablet replica
I20260927 03:26:04.730211 16766 raft_consensus.cc:2229] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:26:04.730322 16766 raft_consensus.cc:2258] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:04.730787 16766 ts_tablet_manager.cc:1905] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:26:04.732678 16766 ts_tablet_manager.cc:1918] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:26:04.732730 16766 log.cc:1196] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/486227f69c254f0bbdcbb4fe64619be2
I20260927 03:26:04.732923 16766 ts_tablet_manager.cc:1939] T 486227f69c254f0bbdcbb4fe64619be2 P 66b09b2401454b29b307204e88c68c31: Deleting consensus metadata
I20260927 03:26:04.733302 16263 catalog_manager.cc:4841] TS 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867): tablet 486227f69c254f0bbdcbb4fe64619be2 (table test-table [id=fa46a46266284ddfbef903a700e86ee3]) successfully deleted
W20260927 03:26:04.735435 16264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42967 (error 108)
I20260927 03:26:04.735697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 16470
I20260927 03:26:04.735891 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.735774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42967 (local address 127.0.0.1:33956)
0927 03:26:04.735824 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:04.735860 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 03:26:04.735992 16264 catalog_manager.cc:4584] TS 344fa21d06a94b5a9ed279793f932e22 (127.0.98.1:42967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 486227f69c254f0bbdcbb4fe64619be2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42967: connect: Connection refused (error 111)
I20260927 03:26:04.740505 16764 ts_tablet_manager.cc:1905] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:26:04.742141 16764 ts_tablet_manager.cc:1918] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:26:04.742316 16764 log.cc:1196] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/486227f69c254f0bbdcbb4fe64619be2
I20260927 03:26:04.742588 16764 ts_tablet_manager.cc:1939] T 486227f69c254f0bbdcbb4fe64619be2 P 62e14b49b72140008596d3632165bd88: Deleting consensus metadata
I20260927 03:26:04.742595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 16606
I20260927 03:26:04.747190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 16244
I20260927 03:26:04.747352 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:04.747209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37929 (local address 127.0.0.1:48414)
0927 03:26:04.747269 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:04.747305 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:37929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 03:26:24.752064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34447
--webserver_interface=127.0.98.62
--webserver_port=42767
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:24.833236 16768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:24.833470 16768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:24.833492 16768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:24.834626 16768 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 03:26:24.834661 16768 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 03:26:24.834671 16768 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 03:26:24.834681 16768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:24.834693 16768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:24.834705 16768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:24.834715 16768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:24.834726 16768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:24.836251 16768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:24.836508 16768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:24.836709 16768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:24.841923 16774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:24.841970 16768 server_base.cc:1034] running on GCE node
W20260927 03:26:24.842032 16776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:24.841956 16773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:24.842517 16768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:24.842804 16768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:24.843994 16768 hybrid_clock.cc:648] HybridClock initialized: now 1790479584843980 us; error 32 us; skew 500 ppm
Sep 27 03:26:24 dist-test-slave-gw2v krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:24.853405 16768 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:26:24.853880 16768 webserver.cc:460] Webserver started at http://127.0.98.62:42767/ using document root <none> and password file <none>
I20260927 03:26:24.854084 16768 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:24.854127 16768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:24.855537 16768 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:26:24.856158 16783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:24.856330 16768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:26:24.856411 16768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "c8c0a65633f045f499723665fdc34fbc"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "c14cf596fbaf9c94d82822fba04e386c"
server_key_iv: "f3b1c923280cef4ed6699a0f10e7bc98"
server_key_version: "encryptionkey@0"
I20260927 03:26:24.856808 16768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:24.863925 16768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:24.865872 16768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:24.865990 16768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:24.869657 16768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34447
I20260927 03:26:24.869707 16835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34447 every 8 connection(s)
I20260927 03:26:24.870056 16768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:24.871629 16836 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:26:24.872368 16836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Bootstrap starting.
I20260927 03:26:24.877162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 16768
I20260927 03:26:24.877472 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42967
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34447
--webserver_port=38795
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:26:24.880908 16836 log.cc:824] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:24.883442 16836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:26:24.883759 16836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Bootstrap complete.
I20260927 03:26:24.885893 16836 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } }
I20260927 03:26:24.886137 16836 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c0a65633f045f499723665fdc34fbc, State: Initialized, Role: FOLLOWER
I20260927 03:26:24.886302 16836 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } }
I20260927 03:26:24.886385 16836 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:24.886438 16836 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:24.886497 16836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 2 FOLLOWER]: Advancing to term 3
I20260927 03:26:24.887622 16836 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } }
I20260927 03:26:24.887737 16836 leader_election.cc:302] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8c0a65633f045f499723665fdc34fbc; no voters:
I20260927 03:26:24.887923 16836 leader_election.cc:288] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 03:26:24.887979 16840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 3 FOLLOWER]: Leader election won for term 3
I20260927 03:26:24.888119 16840 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [term 3 LEADER]: Becoming Leader. State: Replica: c8c0a65633f045f499723665fdc34fbc, State: Running, Role: LEADER
I20260927 03:26:24.888216 16836 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:24.888224 16840 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } }
I20260927 03:26:24.888615 16840 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8c0a65633f045f499723665fdc34fbc. Latest consensus state: current_term: 3 leader_uuid: "c8c0a65633f045f499723665fdc34fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } } }
I20260927 03:26:24.888702 16840 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:24.888667 16842 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c8c0a65633f045f499723665fdc34fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0a65633f045f499723665fdc34fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34447 } } }
I20260927 03:26:24.888994 16842 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:24.889065 16849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:24.889963 16849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:24.890066 16849 catalog_manager.cc:1259] Loaded cluster ID: a6923a81258848978b234a924ee2b699
I20260927 03:26:24.890106 16849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:26:24.891323 16849 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:24.891513 16849 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Loaded TSK: 0
I20260927 03:26:24.891566 16849 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8c0a65633f045f499723665fdc34fbc: Loaded TSK: 1
F20260927 03:26:24.891785 16849 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f00cada89ad google::LogMessage::Fail() at ??:0
@ 0x7f00cadab88a google::LogMessage::SendToLog() at ??:0
@ 0x7f00cada84de google::LogMessage::Flush() at ??:0
@ 0x7f00cada9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f00cd515104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f00cd512d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f00d2d021cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f00d2d0248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f00d2d02594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f00d2cc8c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f00d2cd2ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f00ccc760d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f00ccc6b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f00cdacf6db start_thread at ??:0
@ 0x7f00c98a661f clone at ??:0
W20260927 03:26:24.969460 16838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:24.969697 16838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:24.969736 16838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:24.971112 16838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:24.971158 16838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:24.971199 16838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:26:24.973067 16838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38795
--tserver_master_addrs=127.0.98.62:34447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:24.973332 16838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:24.973560 16838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:24.978675 16877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:24.978694 16880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:24.978718 16878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:24.979097 16838 server_base.cc:1034] running on GCE node
I20260927 03:26:24.979295 16838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:24.979534 16838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:24.980683 16838 hybrid_clock.cc:648] HybridClock initialized: now 1790479584980668 us; error 33 us; skew 500 ppm
Sep 27 03:26:24 dist-test-slave-gw2v krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:24.983274 16838 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:26:24.983748 16838 webserver.cc:460] Webserver started at http://127.0.98.1:38795/ using document root <none> and password file <none>
I20260927 03:26:24.984002 16838 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:24.984053 16838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:24.985324 16838 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:26:24.985925 16887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:24.986088 16838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:24.986167 16838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "344fa21d06a94b5a9ed279793f932e22"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "530e3e60fce0c5342da60805f7f09bdb"
server_key_iv: "5c6c801959963757230cd9c84465a6bb"
server_key_version: "encryptionkey@0"
I20260927 03:26:24.986505 16838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:25.000811 16838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:25.003049 16838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:25.003170 16838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:25.003403 16838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:25.003855 16894 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:26:25.005354 16838 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:26:25.005404 16838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:26:25.005429 16838 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:26:25.006099 16838 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:26:25.006146 16838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:26:25.006217 16894 tablet_bootstrap.cc:492] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22: Bootstrap starting.
I20260927 03:26:25.013567 16838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42967
I20260927 03:26:25.013723 17001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42967 every 8 connection(s)
I20260927 03:26:25.014117 16838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:26:25.017058 16894 log.cc:824] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:25.019630 16894 tablet_bootstrap.cc:492] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:26:25.019927 16894 tablet_bootstrap.cc:492] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22: Bootstrap complete.
I20260927 03:26:25.020678 16894 ts_tablet_manager.cc:1397] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260927 03:26:25.021763 16894 raft_consensus.cc:348] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:25.022008 16894 raft_consensus.cc:729] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 344fa21d06a94b5a9ed279793f932e22, State: Initialized, Role: FOLLOWER
I20260927 03:26:25.022137 16894 consensus_queue.cc:260] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:25.022439 16894 ts_tablet_manager.cc:1428] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:26:25.022873 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 16838
I20260927 03:26:25.023181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45867
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34447
--webserver_port=42145
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:25.089568 17004 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:26:25.015437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:42611)
0927 03:26:25.015671 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:25.015741 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:25.016437 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:25.089354 (+ 72917us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":115,"threads_started":1}
W20260927 03:26:25.089910 17002 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34447 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34447: Client connection negotiation failed: client connection to 127.0.98.62:34447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:26:25.090243 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:25.090124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:37541)
0927 03:26:25.090172 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:25.090219 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 03:26:25.090620 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:25.090535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:46611)
0927 03:26:25.090573 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:25.090600 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 03:26:25.090737 17002 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:26:25.106297 17007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:25.106510 17007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:25.106534 17007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:25.107735 17007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:25.107795 17007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:25.107823 17007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:26:25.109290 17007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42145
--tserver_master_addrs=127.0.98.62:34447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:25.109519 17007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:25.109730 17007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:25.114681 17015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:25.114770 17013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:25.114653 17012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:25.114924 17007 server_base.cc:1034] running on GCE node
I20260927 03:26:25.115156 17007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:25.115454 17007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:25.116603 17007 hybrid_clock.cc:648] HybridClock initialized: now 1790479585116586 us; error 32 us; skew 500 ppm
Sep 27 03:26:25 dist-test-slave-gw2v krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:25.119277 17007 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:26:25.119695 17007 webserver.cc:460] Webserver started at http://127.0.98.2:42145/ using document root <none> and password file <none>
I20260927 03:26:25.119896 17007 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:25.119938 17007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:25.121309 17007 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:25.121822 17022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:25.121980 17007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:25.122061 17007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "66b09b2401454b29b307204e88c68c31"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "469924b985dc159629fc406182f0331e"
server_key_iv: "06a1f6a539187217a7551ae829475bb7"
server_key_version: "encryptionkey@0"
I20260927 03:26:25.122412 17007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:25.136520 17007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:25.138648 17007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:25.138757 17007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:25.138981 17007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:25.139237 17007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:25.139267 17007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:25.139288 17007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:25.139299 17007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:25.144706 17007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45867
I20260927 03:26:25.144788 17135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45867 every 8 connection(s)
I20260927 03:26:25.145105 17007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:26:25.146147 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:25.145769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:42881)
0927 03:26:25.145934 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:25.146006 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
W20260927 03:26:25.146567 17136 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34447 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34447: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
I20260927 03:26:25.146867 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:25.146752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:42973)
0927 03:26:25.146799 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:25.146847 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 03:26:25.147125 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:25.147056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:51135)
0927 03:26:25.147089 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:25.147110 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 03:26:25.147222 17136 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:26:25.148597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 17007
I20260927 03:26:25.148867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37929
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34447
--webserver_port=44131
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:25.228730 17139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:25.228962 17139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:25.228987 17139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:25.230178 17139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:25.230230 17139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:25.230257 17139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:26:25.231799 17139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44131
--tserver_master_addrs=127.0.98.62:34447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:25.232038 17139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:25.232264 17139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:25.237283 17144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:25.237311 17139 server_base.cc:1034] running on GCE node
W20260927 03:26:25.237255 17147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:25.237274 17145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:25.237703 17139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:25.237965 17139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:25.239104 17139 hybrid_clock.cc:648] HybridClock initialized: now 1790479585239073 us; error 50 us; skew 500 ppm
Sep 27 03:26:25 dist-test-slave-gw2v krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:25.241600 17139 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:26:25.242063 17139 webserver.cc:460] Webserver started at http://127.0.98.3:44131/ using document root <none> and password file <none>
I20260927 03:26:25.242260 17139 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:25.242308 17139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:25.243734 17139 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:26:25.244372 17154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:25.244536 17139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:26:25.244623 17139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "62e14b49b72140008596d3632165bd88"
format_stamp: "Formatted at 2026-09-27 03:25:43 on dist-test-slave-gw2v"
server_key: "d8d89862e5b6c60f2f9ffcb3dbd84b67"
server_key_iv: "7c1b9319d5758c0371ca9d523b819b9f"
server_key_version: "encryptionkey@0"
I20260927 03:26:25.244973 17139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:25.284191 17139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:25.286437 17139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:25.286586 17139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:25.286811 17139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:25.287165 17139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:25.287200 17139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:25.287230 17139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:25.287254 17139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:25.293119 17139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37929
I20260927 03:26:25.293180 17267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37929 every 8 connection(s)
I20260927 03:26:25.293543 17139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:26:25.293874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 17139
I20260927 03:26:25.294764 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:25.294330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:57375)
0927 03:26:25.294536 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:25.294612 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
W20260927 03:26:25.295171 17268 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34447 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34447: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
I20260927 03:26:25.295442 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:25.295358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:48421)
0927 03:26:25.295398 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:25.295423 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 03:26:25.295696 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:25.295620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:41863)
0927 03:26:25.295659 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:25.295681 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 03:26:25.295799 17268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:26:26.091336 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:26.091097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:36265)
0927 03:26:26.091239 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:26.091285 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260927 03:26:26.147789 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:26.147572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:48871)
0927 03:26:26.147691 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:26.147729 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260927 03:26:26.296430 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:26.296169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:39685)
0927 03:26:26.296307 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:26.296365 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260927 03:26:26.600636 17275 raft_consensus.cc:482] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:26:26.600788 17275 raft_consensus.cc:504] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:26.601177 17275 leader_election.cc:288] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867), 62e14b49b72140008596d3632165bd88 (127.0.98.3:37929)
Sep 27 03:26:26 dist-test-slave-gw2v krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 03:26:26 dist-test-slave-gw2v krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 03:26:26.611451 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:26.601537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36445 (local address 127.0.98.2:45867)
0927 03:26:26.601732 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:26.601740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:26.601769 (+ 29us) server_negotiation.cc:407] Connection header received
0927 03:26:26.602283 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:26.602285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:26.602326 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:26.602445 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:26.603164 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:26.604022 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:26.605096 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:26.605551 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:26.605657 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:26.607942 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:26.607947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:26.607948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:26.607961 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:26.609756 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:26.610230 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:26.610234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:26.610234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:26.610334 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:26.610551 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:26.610552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:26.610552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:26.610867 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:26.610942 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:26.611210 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 03:26:26.611340 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20260927 03:26:26.611452 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:26.601555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56901 (local address 127.0.98.3:37929)
0927 03:26:26.601723 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:26.601736 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:26.601772 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:26:26.602223 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:26.602225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:26.602268 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:26.602399 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:26.603173 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:26.604031 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:26.605089 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:26.605493 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:26.605592 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:26.608297 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:26.608300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:26.608301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:26.608310 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:26.609983 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:26.610271 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:26.610275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:26.610275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:26.610366 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:26.610553 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:26.610554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:26.610554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:26.610808 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:26.610906 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:26.611214 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 03:26:26.611331 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260927 03:26:26.611469 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:26.601387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37929 (local address 127.0.98.1:56901)
0927 03:26:26.601575 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:26.601587 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:26.601664 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:26.602456 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:26.602459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:26.602848 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:26.603050 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:26.603057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:26.604131 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:26.604132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:26.604997 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:26.604999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:26.605574 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:26.605575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:26.605683 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:26.606290 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:26:26.606294 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:26.608155 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:26.610048 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:26.610049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:26.610050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:26.610202 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:26.610422 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:26.610423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:26.610424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:26.610500 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:26.610886 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:26.610893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:26.611057 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 03:26:26.611145 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 03:26:26.611341 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":69,"thread_start_us":52,"threads_started":1}
I20260927 03:26:26.611801 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:26.601387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45867 (local address 127.0.98.1:36445)
0927 03:26:26.601538 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:26.601552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:26.601648 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:26.602471 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:26.602472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:26.602849 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:26.603051 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:26.603057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:26.604125 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:26.604126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:26.604998 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:26.605000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:26.605632 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:26.605633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:26.605701 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:26.606252 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:26:26.606261 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:26.607821 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:26.609831 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:26.609838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:26.609840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:26.610073 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:26.610395 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:26.610396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:26.610396 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:26.610496 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:26.610934 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:26.610935 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:26.611058 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 03:26:26.611148 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 03:26:26.611412 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":187,"thread_start_us":68,"threads_started":1}
W20260927 03:26:26.612546 16890 leader_election.cc:341] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867): Not found: Tablet not found: 486227f69c254f0bbdcbb4fe64619be2
W20260927 03:26:26.612634 16891 leader_election.cc:341] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 62e14b49b72140008596d3632165bd88 (127.0.98.3:37929): Not found: Tablet not found: 486227f69c254f0bbdcbb4fe64619be2
I20260927 03:26:26.612689 16891 leader_election.cc:302] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 344fa21d06a94b5a9ed279793f932e22; no voters: 62e14b49b72140008596d3632165bd88, 66b09b2401454b29b307204e88c68c31
I20260927 03:26:26.612782 17275 raft_consensus.cc:2726] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:26:27.092053 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:27.091867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:39427)
0927 03:26:27.091955 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:27.092006 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 03:26:27.148630 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:27.148362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:47301)
0927 03:26:27.148514 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:27.148562 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260927 03:26:27.297133 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:27.296940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:42179)
0927 03:26:27.297045 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:27.297082 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":58,"threads_started":1}
I20260927 03:26:28.092810 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:28.092567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:43339)
0927 03:26:28.092714 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:28.092760 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260927 03:26:28.149243 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:28.149054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:38245)
0927 03:26:28.149164 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:28.149200 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260927 03:26:28.297842 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:28.297619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:39255)
0927 03:26:28.297756 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:28.297796 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260927 03:26:28.761646 17285 raft_consensus.cc:482] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:26:28.761744 17285 raft_consensus.cc:504] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:28.761891 17285 leader_election.cc:288] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867), 62e14b49b72140008596d3632165bd88 (127.0.98.3:37929)
W20260927 03:26:28.762367 16890 leader_election.cc:341] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867): Not found: Tablet not found: 486227f69c254f0bbdcbb4fe64619be2
W20260927 03:26:28.762465 16891 leader_election.cc:341] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 62e14b49b72140008596d3632165bd88 (127.0.98.3:37929): Not found: Tablet not found: 486227f69c254f0bbdcbb4fe64619be2
I20260927 03:26:28.762517 16891 leader_election.cc:302] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 344fa21d06a94b5a9ed279793f932e22; no voters: 62e14b49b72140008596d3632165bd88, 66b09b2401454b29b307204e88c68c31
I20260927 03:26:28.762609 17285 raft_consensus.cc:2726] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:26:29.093564 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:29.093318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:33545)
0927 03:26:29.093458 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:29.093504 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260927 03:26:29.149896 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:29.149681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:56157)
0927 03:26:29.149796 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:29.149827 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260927 03:26:29.298550 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:29.298335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:41581)
0927 03:26:29.298464 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:29.298505 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
I20260927 03:26:30.094367 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:30.094100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:58247)
0927 03:26:30.094271 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:30.094317 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260927 03:26:30.150697 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:30.150400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:39539)
0927 03:26:30.150590 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:30.150636 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260927 03:26:30.299335 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:30.299068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:44691)
0927 03:26:30.299242 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:30.299289 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":83,"threads_started":1}
I20260927 03:26:31.095160 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:31.094918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:40655)
0927 03:26:31.095063 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:31.095106 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260927 03:26:31.151467 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:31.151245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:39207)
0927 03:26:31.151373 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:31.151418 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260927 03:26:31.300158 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:31.299905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:39191)
0927 03:26:31.300043 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:31.300088 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260927 03:26:31.561902 17295 raft_consensus.cc:482] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:26:31.561996 17295 raft_consensus.cc:504] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:31.562219 17295 leader_election.cc:288] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867), 62e14b49b72140008596d3632165bd88 (127.0.98.3:37929)
W20260927 03:26:31.562606 16891 leader_election.cc:341] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 62e14b49b72140008596d3632165bd88 (127.0.98.3:37929): Not found: Tablet not found: 486227f69c254f0bbdcbb4fe64619be2
W20260927 03:26:31.562683 16890 leader_election.cc:341] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867): Not found: Tablet not found: 486227f69c254f0bbdcbb4fe64619be2
I20260927 03:26:31.562772 16890 leader_election.cc:302] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 344fa21d06a94b5a9ed279793f932e22; no voters: 62e14b49b72140008596d3632165bd88, 66b09b2401454b29b307204e88c68c31
I20260927 03:26:31.562886 17295 raft_consensus.cc:2726] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:26:32.095956 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:32.095729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:36529)
0927 03:26:32.095860 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:32.095910 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260927 03:26:32.152235 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:32.151919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:38693)
0927 03:26:32.152123 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:32.152177 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260927 03:26:32.300917 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:32.300661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:47457)
0927 03:26:32.300801 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:32.300860 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260927 03:26:33.096720 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:33.096477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:55067)
0927 03:26:33.096631 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:33.096674 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260927 03:26:33.152987 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:33.152692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:56685)
0927 03:26:33.152855 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:33.152913 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260927 03:26:33.301697 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:33.301454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:49781)
0927 03:26:33.301603 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:33.301650 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260927 03:26:33.936935 17302 raft_consensus.cc:482] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:26:33.937088 17302 raft_consensus.cc:504] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:33.937263 17302 leader_election.cc:288] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867), 62e14b49b72140008596d3632165bd88 (127.0.98.3:37929)
W20260927 03:26:33.937672 16891 leader_election.cc:341] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 62e14b49b72140008596d3632165bd88 (127.0.98.3:37929): Not found: Tablet not found: 486227f69c254f0bbdcbb4fe64619be2
W20260927 03:26:33.937779 16890 leader_election.cc:341] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867): Not found: Tablet not found: 486227f69c254f0bbdcbb4fe64619be2
I20260927 03:26:33.937850 16890 leader_election.cc:302] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 344fa21d06a94b5a9ed279793f932e22; no voters: 62e14b49b72140008596d3632165bd88, 66b09b2401454b29b307204e88c68c31
I20260927 03:26:33.937947 17302 raft_consensus.cc:2726] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:26:34.097497 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:34.097223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:53559)
0927 03:26:34.097355 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:34.097398 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260927 03:26:34.153715 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:34.153500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:39031)
0927 03:26:34.153623 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:34.153669 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260927 03:26:34.302474 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:34.302185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:41645)
0927 03:26:34.302317 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:34.302425 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260927 03:26:35.098362 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:35.098072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:53481)
0927 03:26:35.098213 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:35.098259 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260927 03:26:35.154438 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:35.154195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:59599)
0927 03:26:35.154350 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:35.154389 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
I20260927 03:26:35.303198 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:35.302980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:37097)
0927 03:26:35.303115 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:35.303153 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260927 03:26:36.099291 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:36.098955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:40197)
0927 03:26:36.099192 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:36.099241 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260927 03:26:36.155104 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:36.154877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:60911)
0927 03:26:36.155019 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:36.155058 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260927 03:26:36.303946 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:36.303717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:47903)
0927 03:26:36.303856 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:36.303899 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260927 03:26:36.429462 17312 raft_consensus.cc:482] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:26:36.429550 17312 raft_consensus.cc:504] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b09b2401454b29b307204e88c68c31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } } peers { permanent_uuid: "62e14b49b72140008596d3632165bd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37929 } } peers { permanent_uuid: "344fa21d06a94b5a9ed279793f932e22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42967 } }
I20260927 03:26:36.429695 17312 leader_election.cc:288] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867), 62e14b49b72140008596d3632165bd88 (127.0.98.3:37929)
W20260927 03:26:36.430034 16890 leader_election.cc:341] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66b09b2401454b29b307204e88c68c31 (127.0.98.2:45867): Not found: Tablet not found: 486227f69c254f0bbdcbb4fe64619be2
W20260927 03:26:36.430135 16891 leader_election.cc:341] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 62e14b49b72140008596d3632165bd88 (127.0.98.3:37929): Not found: Tablet not found: 486227f69c254f0bbdcbb4fe64619be2
I20260927 03:26:36.430177 16891 leader_election.cc:302] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 344fa21d06a94b5a9ed279793f932e22; no voters: 62e14b49b72140008596d3632165bd88, 66b09b2401454b29b307204e88c68c31
I20260927 03:26:36.430243 17312 raft_consensus.cc:2726] T 486227f69c254f0bbdcbb4fe64619be2 P 344fa21d06a94b5a9ed279793f932e22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:26:37.100031 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:37.099822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:34611)
0927 03:26:37.099939 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:37.099983 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260927 03:26:37.155952 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:37.155624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:54481)
0927 03:26:37.155813 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:37.155892 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
I20260927 03:26:37.304593 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:37.304343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:39733)
0927 03:26:37.304483 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:37.304541 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260927 03:26:38.100756 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:38.100522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:39425)
0927 03:26:38.100664 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:38.100708 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260927 03:26:38.156736 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:38.156501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:35795)
0927 03:26:38.156632 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:38.156688 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260927 03:26:38.305363 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:38.305131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:45011)
0927 03:26:38.305269 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:38.305316 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260927 03:26:39.101480 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:39.101275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:41839)
0927 03:26:39.101400 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:39.101431 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260927 03:26:39.157526 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:39.157240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:34789)
0927 03:26:39.157434 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:39.157479 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":85,"threads_started":1}
I20260927 03:26:39.306205 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:39.305953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:42325)
0927 03:26:39.306105 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:39.306158 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260927 03:26:40.102427 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:40.102167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.1:44497)
0927 03:26:40.102336 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:40.102382 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260927 03:26:40.158342 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:40.158089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.2:33355)
0927 03:26:40.158249 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:40.158291 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260927 03:26:40.304437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 16838
I20260927 03:26:40.306985 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:40.306774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34447 (local address 127.0.98.3:46755)
0927 03:26:40.306903 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:40.306942 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260927 03:26:40.308862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 17007
I20260927 03:26:40.313764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 17139
2026-09-27T03:26:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58159 ms)
[----------] 30 tests from SecurityITest (231093 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:26:40 dist-test-slave-gw2v krb5kdc[17327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:26:40 dist-test-slave-gw2v krb5kdc[17327](info): set up 2 sockets
Sep 27 03:26:40 dist-test-slave-gw2v krb5kdc[17327](info): commencing operation
krb5kdc: starting...
W20260927 03:26:41.274466 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:26:41 dist-test-slave-gw2v krb5kdc[17327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:26:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:26:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:26:41.394008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46789
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:41.476516 17340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:41.476729 17340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:41.476759 17340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:41.477886 17340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:41.477928 17340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:41.477946 17340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:41.477960 17340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:41.477977 17340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:41.479487 17340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:41.479727 17340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:41.479946 17340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:41.484467 17348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:41.484467 17345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:41.484488 17346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:41.484835 17340 server_base.cc:1034] running on GCE node
I20260927 03:26:41.485049 17340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:41.485316 17340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:41.486459 17340 hybrid_clock.cc:648] HybridClock initialized: now 1790479601486443 us; error 27 us; skew 500 ppm
Sep 27 03:26:41 dist-test-slave-gw2v krb5kdc[17327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:41.488826 17340 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:26:41.489254 17340 webserver.cc:460] Webserver started at http://127.0.98.62:42319/ using document root <none> and password file <none>
I20260927 03:26:41.489440 17340 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:41.489480 17340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:41.489570 17340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:41.490557 17340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "3ca9268473de4453b880cf7fcd5a896f"
format_stamp: "Formatted at 2026-09-27 03:26:41 on dist-test-slave-gw2v"
server_key: "d2a30c2a3fee929b37be6ca675d86d4b"
server_key_iv: "4592ef21e6c4f38ba4d180cd3c0fe6ce"
server_key_version: "encryptionkey@0"
I20260927 03:26:41.490890 17340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ca9268473de4453b880cf7fcd5a896f"
format_stamp: "Formatted at 2026-09-27 03:26:41 on dist-test-slave-gw2v"
server_key: "d2a30c2a3fee929b37be6ca675d86d4b"
server_key_iv: "4592ef21e6c4f38ba4d180cd3c0fe6ce"
server_key_version: "encryptionkey@0"
I20260927 03:26:41.492082 17340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:41.492806 17355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:41.492957 17340 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:26:41.493027 17340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "3ca9268473de4453b880cf7fcd5a896f"
format_stamp: "Formatted at 2026-09-27 03:26:41 on dist-test-slave-gw2v"
server_key: "d2a30c2a3fee929b37be6ca675d86d4b"
server_key_iv: "4592ef21e6c4f38ba4d180cd3c0fe6ce"
server_key_version: "encryptionkey@0"
I20260927 03:26:41.493096 17340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:41.504848 17340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:41.507171 17340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:41.507299 17340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:41.511081 17340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46789
I20260927 03:26:41.511124 17407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46789 every 8 connection(s)
I20260927 03:26:41.511447 17340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:41.512223 17408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:41.514442 17408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f: Bootstrap starting.
I20260927 03:26:41.515017 17408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:41.515280 17408 log.cc:824] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:41.515926 17408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f: No bootstrap required, opened a new log
I20260927 03:26:41.517518 17408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9268473de4453b880cf7fcd5a896f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46789 } }
I20260927 03:26:41.517655 17408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:41.517678 17408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ca9268473de4453b880cf7fcd5a896f, State: Initialized, Role: FOLLOWER
I20260927 03:26:41.517812 17408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9268473de4453b880cf7fcd5a896f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46789 } }
I20260927 03:26:41.517881 17408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:41.517903 17408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:41.517951 17408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:41.518492 17408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9268473de4453b880cf7fcd5a896f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46789 } }
I20260927 03:26:41.518577 17408 leader_election.cc:302] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ca9268473de4453b880cf7fcd5a896f; no voters:
I20260927 03:26:41.518754 17408 leader_election.cc:288] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:26:41.518797 17411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:41.518859 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 17340
I20260927 03:26:41.518970 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:26:41.519011 17411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [term 1 LEADER]: Becoming Leader. State: Replica: 3ca9268473de4453b880cf7fcd5a896f, State: Running, Role: LEADER
I20260927 03:26:41.519039 17408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:41.519140 392 external_mini_cluster.cc:1428] Setting key f889260015c4b8b11d94468c5ff24761
I20260927 03:26:41.519147 17411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9268473de4453b880cf7fcd5a896f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46789 } }
I20260927 03:26:41.519687 17412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ca9268473de4453b880cf7fcd5a896f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9268473de4453b880cf7fcd5a896f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46789 } } }
I20260927 03:26:41.519781 17412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:41.519984 17413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ca9268473de4453b880cf7fcd5a896f. Latest consensus state: current_term: 1 leader_uuid: "3ca9268473de4453b880cf7fcd5a896f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9268473de4453b880cf7fcd5a896f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46789 } } }
I20260927 03:26:41.520098 17413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:41.520107 17419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:41.520666 17419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:41.522416 17419 catalog_manager.cc:1347] Generated new cluster ID: 13782e461f3e4f9bad8713524cf35bde
I20260927 03:26:41.522472 17419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:26:41 dist-test-slave-gw2v krb5kdc[17327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:26:41.530248 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:41.519757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.0.98.62:46789)
0927 03:26:41.520742 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:41.520752 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:41.520789 (+ 37us) server_negotiation.cc:407] Connection header received
0927 03:26:41.520919 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:41.520927 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:41.520990 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:41.521137 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:41.521880 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:41.523000 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:41.523755 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:41.524203 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:41.524309 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:41.526480 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:41.526485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:41.526486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:41.526499 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:41.528724 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:41.529089 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:41.529093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:41.529093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:41.529203 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:41.529391 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:41.529392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:41.529392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:41.529648 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:41.529812 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:41.529885 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 03:26:41.530033 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":933,"thread_start_us":77,"threads_started":1}
I20260927 03:26:41.532611 17419 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:26:41.533373 17419 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:41.539544 17419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f: Generated new TSK 0
I20260927 03:26:41.539773 17419 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:26:41.571863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46789
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:41.677661 17434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:41.677898 17434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:41.677934 17434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:41.679312 17434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:41.679366 17434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:41.679405 17434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:26:41.681030 17434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:41.681308 17434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:41.681536 17434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:41.687496 17440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:41.687530 17442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:41.687539 17439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:41.688604 17434 server_base.cc:1034] running on GCE node
I20260927 03:26:41.688819 17434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:41.689127 17434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:41.690287 17434 hybrid_clock.cc:648] HybridClock initialized: now 1790479601690281 us; error 34 us; skew 500 ppm
Sep 27 03:26:41 dist-test-slave-gw2v krb5kdc[17327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:41.692651 17434 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:26:41.693104 17434 webserver.cc:460] Webserver started at http://127.0.98.1:43669/ using document root <none> and password file <none>
I20260927 03:26:41.693307 17434 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:41.693354 17434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:41.693476 17434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:41.694542 17434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "421dd83aade549fc913e87ca7629e042"
format_stamp: "Formatted at 2026-09-27 03:26:41 on dist-test-slave-gw2v"
server_key: "e47ecf423ee8c19ba8017b82b12e85ad"
server_key_iv: "97c54c6838a044746d4148e459649bb2"
server_key_version: "encryptionkey@0"
I20260927 03:26:41.694917 17434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "421dd83aade549fc913e87ca7629e042"
format_stamp: "Formatted at 2026-09-27 03:26:41 on dist-test-slave-gw2v"
server_key: "e47ecf423ee8c19ba8017b82b12e85ad"
server_key_iv: "97c54c6838a044746d4148e459649bb2"
server_key_version: "encryptionkey@0"
I20260927 03:26:41.696352 17434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:26:41.697122 17449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:41.697290 17434 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:26:41.697366 17434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "421dd83aade549fc913e87ca7629e042"
format_stamp: "Formatted at 2026-09-27 03:26:41 on dist-test-slave-gw2v"
server_key: "e47ecf423ee8c19ba8017b82b12e85ad"
server_key_iv: "97c54c6838a044746d4148e459649bb2"
server_key_version: "encryptionkey@0"
I20260927 03:26:41.697419 17434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:41.711601 17434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:41.713402 17434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:41.713521 17434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:41.713709 17434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:41.713977 17434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:41.714007 17434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:41.714028 17434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:41.714044 17434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:41.720014 17434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40537
I20260927 03:26:41.720065 17562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40537 every 8 connection(s)
I20260927 03:26:41.720391 17434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:26:41 dist-test-slave-gw2v krb5kdc[17327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:26:41.727509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 17434
I20260927 03:26:41.727600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:26:41.727773 392 external_mini_cluster.cc:1428] Setting key ce54e56814c2ebb1822b51a89b04af87
I20260927 03:26:41.729657 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:41.721054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39981 (local address 127.0.98.62:46789)
0927 03:26:41.721106 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:41.721108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:41.721341 (+ 233us) server_negotiation.cc:407] Connection header received
0927 03:26:41.722319 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:41.722321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:41.722336 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:41.722396 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:41.723052 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:41.723567 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:41.724557 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:41.724757 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:41.724808 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:41.726990 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:41.726992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:41.726992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:41.726998 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:41.728439 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:41.728801 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:41.728802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:41.728803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:41.728866 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:41.729070 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:41.729070 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:41.729071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:41.729194 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:41.729257 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:41.729477 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 03:26:41.729562 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:26:41.729836 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:41.721059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46789 (local address 127.0.98.1:39981)
0927 03:26:41.721208 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:41.721246 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:41.722200 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:41.722464 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:41.722468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:41.722731 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:41.722979 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:41.722984 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:41.723632 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:41.723632 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:41.724490 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:41.724493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:41.724846 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:41.724847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:41.724949 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:41.725450 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:26:41.725458 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:41.726897 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:41.728509 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:41.728516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:41.728517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:41.728735 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:41.728927 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:41.728927 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:41.728927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:41.729023 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:41.729246 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:41.729249 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:41.729369 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 03:26:41.729445 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 03:26:41.729585 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260927 03:26:41.730623 17563 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46789
I20260927 03:26:41.730712 17563 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:41.730921 17563 heartbeater.cc:511] Master 127.0.98.62:46789 requested a full tablet report, sending...
I20260927 03:26:41.731436 17372 ts_manager.cc:194] Registered new tserver with Master: 421dd83aade549fc913e87ca7629e042 (127.0.98.1:40537)
I20260927 03:26:41.732057 17372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39981
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:26:41.754137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46789
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:41.833268 17568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:41.833484 17568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:41.833518 17568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:41.834720 17568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:41.834766 17568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:41.834801 17568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:26:41.836386 17568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:41.836633 17568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:41.836846 17568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:41.842026 17573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:41.842061 17576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:41.842026 17574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:41.842046 17568 server_base.cc:1034] running on GCE node
I20260927 03:26:41.842427 17568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:41.842674 17568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:41.843900 17568 hybrid_clock.cc:648] HybridClock initialized: now 1790479601843897 us; error 46 us; skew 500 ppm
Sep 27 03:26:41 dist-test-slave-gw2v krb5kdc[17327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:41.846545 17568 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:26:41.847090 17568 webserver.cc:460] Webserver started at http://127.0.98.2:38737/ using document root <none> and password file <none>
I20260927 03:26:41.847309 17568 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:41.847357 17568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:41.847477 17568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:41.848565 17568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "3043751cae6f4656b9f93278a61a08ae"
format_stamp: "Formatted at 2026-09-27 03:26:41 on dist-test-slave-gw2v"
server_key: "e4817b79e7d86e0c273e1ba151cd30d3"
server_key_iv: "e4f7efff922485af5d3172f89945e2b0"
server_key_version: "encryptionkey@0"
I20260927 03:26:41.848891 17568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3043751cae6f4656b9f93278a61a08ae"
format_stamp: "Formatted at 2026-09-27 03:26:41 on dist-test-slave-gw2v"
server_key: "e4817b79e7d86e0c273e1ba151cd30d3"
server_key_iv: "e4f7efff922485af5d3172f89945e2b0"
server_key_version: "encryptionkey@0"
I20260927 03:26:41.850109 17568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:26:41.850919 17583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:41.851061 17568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:41.851148 17568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "3043751cae6f4656b9f93278a61a08ae"
format_stamp: "Formatted at 2026-09-27 03:26:41 on dist-test-slave-gw2v"
server_key: "e4817b79e7d86e0c273e1ba151cd30d3"
server_key_iv: "e4f7efff922485af5d3172f89945e2b0"
server_key_version: "encryptionkey@0"
I20260927 03:26:41.851210 17568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:41.858707 17568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:41.860443 17568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:41.860570 17568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:41.860769 17568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:41.861038 17568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:41.861078 17568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:41.861114 17568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:41.861136 17568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:41.866485 17568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43699
I20260927 03:26:41.866540 17696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43699 every 8 connection(s)
I20260927 03:26:41.866859 17568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:26:41.868563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 17568
I20260927 03:26:41.868674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:26:41.868873 392 external_mini_cluster.cc:1428] Setting key ceab5153cdf244260d14318b7be71af9
Sep 27 03:26:41 dist-test-slave-gw2v krb5kdc[17327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:26:41.877032 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:41.867542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59869 (local address 127.0.98.62:46789)
0927 03:26:41.867612 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:41.867614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:41.867856 (+ 242us) server_negotiation.cc:407] Connection header received
0927 03:26:41.869070 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:41.869072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:41.869086 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:41.869161 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:41.869958 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:41.870467 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:41.871692 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:41.871912 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:41.871979 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:41.874344 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:41.874346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:41.874347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:41.874353 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:41.875657 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:41.876104 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:41.876104 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:41.876105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:41.876153 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:41.876423 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:41.876423 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:41.876423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:41.876528 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:41.876572 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:41.876897 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 03:26:41.876964 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 03:26:41.877300 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:41.867496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46789 (local address 127.0.98.2:59869)
0927 03:26:41.867723 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:41.867771 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:41.868907 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:41.869249 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:41.869254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:41.869563 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:41.869842 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:41.869849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:41.870556 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:41.870557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:41.871604 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:41.871607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:41.872023 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:41.872025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:41.872152 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:41.872716 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:26:41.872724 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:41.874243 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:41.875737 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:41.875745 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:41.875747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:41.876035 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:41.876210 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:41.876211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:41.876211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:41.876360 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:41.876584 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:41.876590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:41.876767 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 03:26:41.876884 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 03:26:41.877048 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260927 03:26:41.877825 17697 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46789
I20260927 03:26:41.877923 17697 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:41.878078 17697 heartbeater.cc:511] Master 127.0.98.62:46789 requested a full tablet report, sending...
I20260927 03:26:41.878489 17372 ts_manager.cc:194] Registered new tserver with Master: 3043751cae6f4656b9f93278a61a08ae (127.0.98.2:43699)
I20260927 03:26:41.878947 17372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:26:41.894274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46789
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:41.976550 17702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:41.976768 17702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:41.976791 17702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:41.977948 17702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:41.977990 17702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:41.978015 17702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:26:41.979511 17702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:41.979701 17702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:41.979875 17702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:41.984670 17708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:41.984781 17710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:41.984670 17707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:41.984853 17702 server_base.cc:1034] running on GCE node
I20260927 03:26:41.985114 17702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:41.985368 17702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:41.986523 17702 hybrid_clock.cc:648] HybridClock initialized: now 1790479601986507 us; error 27 us; skew 500 ppm
Sep 27 03:26:41 dist-test-slave-gw2v krb5kdc[17327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:41.989125 17702 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:26:41.989526 17702 webserver.cc:460] Webserver started at http://127.0.98.3:44401/ using document root <none> and password file <none>
I20260927 03:26:41.989730 17702 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:41.989770 17702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:41.989873 17702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:41.991000 17702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f"
format_stamp: "Formatted at 2026-09-27 03:26:41 on dist-test-slave-gw2v"
server_key: "6a59d8b2766aef9511ff28f19b153458"
server_key_iv: "826a97e87815ffdec81a834a3696aa9c"
server_key_version: "encryptionkey@0"
I20260927 03:26:41.991304 17702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f"
format_stamp: "Formatted at 2026-09-27 03:26:41 on dist-test-slave-gw2v"
server_key: "6a59d8b2766aef9511ff28f19b153458"
server_key_iv: "826a97e87815ffdec81a834a3696aa9c"
server_key_version: "encryptionkey@0"
I20260927 03:26:41.992496 17702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:41.993330 17717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:41.993515 17702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:41.993655 17702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f"
format_stamp: "Formatted at 2026-09-27 03:26:41 on dist-test-slave-gw2v"
server_key: "6a59d8b2766aef9511ff28f19b153458"
server_key_iv: "826a97e87815ffdec81a834a3696aa9c"
server_key_version: "encryptionkey@0"
I20260927 03:26:41.993716 17702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:42.006531 17702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:42.008870 17702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:42.008989 17702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:42.009209 17702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:42.009485 17702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:42.009516 17702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:42.009536 17702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:42.009552 17702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:42.014806 17702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36273
I20260927 03:26:42.014885 17830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36273 every 8 connection(s)
I20260927 03:26:42.015192 17702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:26:42.019359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 17702
I20260927 03:26:42.019465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:26:42.019650 392 external_mini_cluster.cc:1428] Setting key 4073f2985c40c5bf3bd502dbb13f1e72
Sep 27 03:26:42 dist-test-slave-gw2v krb5kdc[17327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:26:42.025326 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.015903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39253 (local address 127.0.98.62:46789)
0927 03:26:42.015978 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.015981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.016118 (+ 137us) server_negotiation.cc:407] Connection header received
0927 03:26:42.017126 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.017127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.017139 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.017204 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:42.017993 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.018498 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.019935 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.020242 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.020317 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.022614 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:42.022615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:42.022615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:42.022622 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:42.023986 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:42.024422 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:42.024422 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:42.024423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:42.024475 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:42.024747 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:42.024748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:42.024749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:42.024870 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:42.024940 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.025187 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.025259 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 03:26:42.025527 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.015865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46789 (local address 127.0.98.3:39253)
0927 03:26:42.016004 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:42.016041 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:42.017000 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:42.017314 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:42.017319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:42.017662 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:42.017877 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.017886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.018615 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.018625 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.019833 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.019836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.020325 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.020326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.020483 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.021004 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:26:42.021012 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:42.022505 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:42.024071 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:42.024077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:42.024079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:42.024347 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:42.024543 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:42.024544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:42.024545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:42.024664 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:42.024940 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:42.024946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:42.025076 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 03:26:42.025161 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:26:42.025300 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260927 03:26:42.025974 17831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46789
I20260927 03:26:42.026072 17831 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:42.026274 17831 heartbeater.cc:511] Master 127.0.98.62:46789 requested a full tablet report, sending...
I20260927 03:26:42.026640 17372 ts_manager.cc:194] Registered new tserver with Master: 9dc9aaa73e244124b3cb7130cc2f9e5f (127.0.98.3:36273)
I20260927 03:26:42.027040 17372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39253
I20260927 03:26:42.031584 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:26:42.038465 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.032909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.0.98.62:46789)
0927 03:26:42.032960 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.032962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.033056 (+ 94us) server_negotiation.cc:407] Connection header received
0927 03:26:42.033120 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.033120 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.033131 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.033180 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:42.033725 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.034208 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.034917 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.035100 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.035161 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.036236 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:42.036237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:42.036238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:42.036245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:42.037439 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:42.037732 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:42.037733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:42.037734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:42.037782 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:42.037935 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:42.037936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:42.037937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:42.038079 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:42.038165 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.038303 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.038376 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:26:42.040262 17372 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:26:42.041878 17372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:26:42.048323 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.044301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56110 (local address 127.0.98.3:36273)
0927 03:26:42.044501 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.044506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.044704 (+ 198us) server_negotiation.cc:407] Connection header received
0927 03:26:42.044752 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.044754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.044768 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.044902 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:42.045080 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.046210 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.047456 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.047930 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.047994 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.048055 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.048201 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.048240 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
I20260927 03:26:42.048316 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.044185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36273 (local address 127.0.0.1:56110)
0927 03:26:42.044534 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:42.044544 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:42.044627 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:42.044919 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:42.044923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:42.044925 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:42.044999 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.045001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.046319 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.046320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.047391 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.047393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.048003 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.048004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.048089 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.048104 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:26:42.048180 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 03:26:42.048234 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":14,"thread_start_us":81,"threads_started":1}
I20260927 03:26:42.048555 17845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.044388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43699 (local address 127.0.0.1:45638)
0927 03:26:42.044545 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:42.044553 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:42.044601 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:42.044966 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:42.044967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:42.044969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:42.045027 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.045029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.045935 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.045936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.047174 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.047177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.048157 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.048158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.048220 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.048224 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:26:42.048268 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:26:42.048311 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260927 03:26:42.048636 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.044380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.0.98.1:40537)
0927 03:26:42.044717 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.044721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.044732 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:26:42.044787 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.044789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.044805 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.045086 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:42.045280 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.046324 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.047475 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.048166 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.048233 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.048285 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.048523 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.048564 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":76,"threads_started":1}
I20260927 03:26:42.048666 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.044184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40537 (local address 127.0.0.1:35896)
0927 03:26:42.044335 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:42.044377 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:42.044439 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:42.045168 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:42.045169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:42.045171 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:42.045202 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.045203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.046420 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.046421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.047374 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.047377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.048315 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.048315 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.048367 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.048370 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:42.048413 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:26:42.048453 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":3,"thread_start_us":50,"threads_started":1}
I20260927 03:26:42.049067 17765 tablet_service.cc:1467] Processing CreateTablet for tablet c6081cf2fcab43dc8712b21f5bba56a3 (DEFAULT_TABLE table=test-table [id=de697610ad9e47cea3a3b6e3f5732118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:42.049355 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.044584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.0.98.2:43699)
0927 03:26:42.044751 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.044755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.044762 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:26:42.044796 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.044798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.044811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.044878 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:42.045103 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.045847 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.047256 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.047995 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.048065 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.048121 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.048354 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.049015 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20260927 03:26:42.049443 17765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6081cf2fcab43dc8712b21f5bba56a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:42.049341 17497 tablet_service.cc:1467] Processing CreateTablet for tablet c6081cf2fcab43dc8712b21f5bba56a3 (DEFAULT_TABLE table=test-table [id=de697610ad9e47cea3a3b6e3f5732118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:42.049649 17497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6081cf2fcab43dc8712b21f5bba56a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:42.049911 17631 tablet_service.cc:1467] Processing CreateTablet for tablet c6081cf2fcab43dc8712b21f5bba56a3 (DEFAULT_TABLE table=test-table [id=de697610ad9e47cea3a3b6e3f5732118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:42.050184 17631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6081cf2fcab43dc8712b21f5bba56a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:42.052789 17848 tablet_bootstrap.cc:492] T c6081cf2fcab43dc8712b21f5bba56a3 P 421dd83aade549fc913e87ca7629e042: Bootstrap starting.
I20260927 03:26:42.052960 17849 tablet_bootstrap.cc:492] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f: Bootstrap starting.
I20260927 03:26:42.053395 17850 tablet_bootstrap.cc:492] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae: Bootstrap starting.
I20260927 03:26:42.053550 17849 tablet_bootstrap.cc:654] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:42.053558 17848 tablet_bootstrap.cc:654] T c6081cf2fcab43dc8712b21f5bba56a3 P 421dd83aade549fc913e87ca7629e042: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:42.054062 17848 log.cc:824] T c6081cf2fcab43dc8712b21f5bba56a3 P 421dd83aade549fc913e87ca7629e042: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:42.054092 17849 log.cc:824] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:42.054136 17850 tablet_bootstrap.cc:654] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:42.054656 17850 log.cc:824] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:42.054729 17848 tablet_bootstrap.cc:492] T c6081cf2fcab43dc8712b21f5bba56a3 P 421dd83aade549fc913e87ca7629e042: No bootstrap required, opened a new log
I20260927 03:26:42.054787 17848 ts_tablet_manager.cc:1397] T c6081cf2fcab43dc8712b21f5bba56a3 P 421dd83aade549fc913e87ca7629e042: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:26:42.054862 17849 tablet_bootstrap.cc:492] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f: No bootstrap required, opened a new log
I20260927 03:26:42.054930 17849 ts_tablet_manager.cc:1397] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:26:42.055462 17850 tablet_bootstrap.cc:492] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae: No bootstrap required, opened a new log
I20260927 03:26:42.055578 17850 ts_tablet_manager.cc:1397] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 03:26:42.056488 17848 raft_consensus.cc:348] T c6081cf2fcab43dc8712b21f5bba56a3 P 421dd83aade549fc913e87ca7629e042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421dd83aade549fc913e87ca7629e042" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36273 } } peers { permanent_uuid: "3043751cae6f4656b9f93278a61a08ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } }
I20260927 03:26:42.056638 17848 raft_consensus.cc:374] T c6081cf2fcab43dc8712b21f5bba56a3 P 421dd83aade549fc913e87ca7629e042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:42.056671 17848 raft_consensus.cc:729] T c6081cf2fcab43dc8712b21f5bba56a3 P 421dd83aade549fc913e87ca7629e042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 421dd83aade549fc913e87ca7629e042, State: Initialized, Role: FOLLOWER
I20260927 03:26:42.056810 17848 consensus_queue.cc:260] T c6081cf2fcab43dc8712b21f5bba56a3 P 421dd83aade549fc913e87ca7629e042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421dd83aade549fc913e87ca7629e042" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36273 } } peers { permanent_uuid: "3043751cae6f4656b9f93278a61a08ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } }
I20260927 03:26:42.056799 17849 raft_consensus.cc:348] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421dd83aade549fc913e87ca7629e042" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36273 } } peers { permanent_uuid: "3043751cae6f4656b9f93278a61a08ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } }
I20260927 03:26:42.056972 17849 raft_consensus.cc:374] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:42.057014 17849 raft_consensus.cc:729] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc9aaa73e244124b3cb7130cc2f9e5f, State: Initialized, Role: FOLLOWER
I20260927 03:26:42.057114 17848 ts_tablet_manager.cc:1428] T c6081cf2fcab43dc8712b21f5bba56a3 P 421dd83aade549fc913e87ca7629e042: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:26:42.057120 17849 consensus_queue.cc:260] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421dd83aade549fc913e87ca7629e042" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36273 } } peers { permanent_uuid: "3043751cae6f4656b9f93278a61a08ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } }
I20260927 03:26:42.057237 17563 heartbeater.cc:503] Master 127.0.98.62:46789 was elected leader, sending a full tablet report...
I20260927 03:26:42.057384 17849 ts_tablet_manager.cc:1428] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:26:42.057401 17831 heartbeater.cc:503] Master 127.0.98.62:46789 was elected leader, sending a full tablet report...
I20260927 03:26:42.057587 17850 raft_consensus.cc:348] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421dd83aade549fc913e87ca7629e042" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36273 } } peers { permanent_uuid: "3043751cae6f4656b9f93278a61a08ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } }
I20260927 03:26:42.057746 17850 raft_consensus.cc:374] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:42.057782 17850 raft_consensus.cc:729] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3043751cae6f4656b9f93278a61a08ae, State: Initialized, Role: FOLLOWER
I20260927 03:26:42.057878 17850 consensus_queue.cc:260] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421dd83aade549fc913e87ca7629e042" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36273 } } peers { permanent_uuid: "3043751cae6f4656b9f93278a61a08ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } }
I20260927 03:26:42.058151 17850 ts_tablet_manager.cc:1428] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:26:42.058271 17697 heartbeater.cc:503] Master 127.0.98.62:46789 was elected leader, sending a full tablet report...
W20260927 03:26:42.117350 17698 tablet.cc:2369] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:26:42.212448 17856 raft_consensus.cc:482] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:26:42.212579 17856 raft_consensus.cc:504] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421dd83aade549fc913e87ca7629e042" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36273 } } peers { permanent_uuid: "3043751cae6f4656b9f93278a61a08ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } }
I20260927 03:26:42.212890 17856 leader_election.cc:288] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 421dd83aade549fc913e87ca7629e042 (127.0.98.1:40537), 9dc9aaa73e244124b3cb7130cc2f9e5f (127.0.98.3:36273)
I20260927 03:26:42.215842 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.213158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46669 (local address 127.0.98.1:40537)
0927 03:26:42.213227 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.213229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.213238 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:26:42.213267 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.213267 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.213280 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.213343 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:42.213649 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.214164 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.215174 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.215535 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.215595 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.215606 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.215735 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.215770 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 03:26:42.215854 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.213180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46843 (local address 127.0.98.3:36273)
0927 03:26:42.213252 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.213253 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.213506 (+ 253us) server_negotiation.cc:407] Connection header received
0927 03:26:42.213545 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.213546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.213555 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.213603 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:42.213731 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.214232 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.215192 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.215548 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.215603 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.215612 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.215755 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.215790 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 03:26:42.215853 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.213078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40537 (local address 127.0.98.2:46669)
0927 03:26:42.213129 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:42.213140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:42.213183 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:42.213354 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:42.213355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:42.213358 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:42.213436 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.213439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.214246 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.214248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.215096 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.215099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.215603 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.215603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.215677 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.215680 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:42.215725 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:26:42.215766 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":1}
I20260927 03:26:42.215864 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.213078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36273 (local address 127.0.98.2:46843)
0927 03:26:42.213427 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:42.213437 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:42.213500 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:42.213608 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:42.213608 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:42.213610 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:42.213657 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.213658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.214313 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.214315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.215107 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.215109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.215615 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.215616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.215686 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.215689 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:42.215739 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:26:42.215776 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":151,"thread_start_us":127,"threads_started":1}
I20260927 03:26:42.216188 17517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6081cf2fcab43dc8712b21f5bba56a3" candidate_uuid: "3043751cae6f4656b9f93278a61a08ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "421dd83aade549fc913e87ca7629e042" is_pre_election: true
I20260927 03:26:42.216390 17517 raft_consensus.cc:2454] T c6081cf2fcab43dc8712b21f5bba56a3 P 421dd83aade549fc913e87ca7629e042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3043751cae6f4656b9f93278a61a08ae in term 0.
I20260927 03:26:42.216391 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6081cf2fcab43dc8712b21f5bba56a3" candidate_uuid: "3043751cae6f4656b9f93278a61a08ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f" is_pre_election: true
I20260927 03:26:42.216531 17785 raft_consensus.cc:2454] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3043751cae6f4656b9f93278a61a08ae in term 0.
I20260927 03:26:42.216624 17584 leader_election.cc:302] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3043751cae6f4656b9f93278a61a08ae, 421dd83aade549fc913e87ca7629e042; no voters:
I20260927 03:26:42.216768 17856 raft_consensus.cc:2781] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:26:42.216823 17856 raft_consensus.cc:482] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:26:42.216838 17856 raft_consensus.cc:3037] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:42.217476 17856 raft_consensus.cc:504] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421dd83aade549fc913e87ca7629e042" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36273 } } peers { permanent_uuid: "3043751cae6f4656b9f93278a61a08ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } }
I20260927 03:26:42.217590 17856 leader_election.cc:288] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [CANDIDATE]: Term 1 election: Requested vote from peers 421dd83aade549fc913e87ca7629e042 (127.0.98.1:40537), 9dc9aaa73e244124b3cb7130cc2f9e5f (127.0.98.3:36273)
I20260927 03:26:42.217764 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6081cf2fcab43dc8712b21f5bba56a3" candidate_uuid: "3043751cae6f4656b9f93278a61a08ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f"
I20260927 03:26:42.217834 17785 raft_consensus.cc:3037] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:42.217816 17517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6081cf2fcab43dc8712b21f5bba56a3" candidate_uuid: "3043751cae6f4656b9f93278a61a08ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "421dd83aade549fc913e87ca7629e042"
I20260927 03:26:42.217896 17517 raft_consensus.cc:3037] T c6081cf2fcab43dc8712b21f5bba56a3 P 421dd83aade549fc913e87ca7629e042 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:42.218502 17785 raft_consensus.cc:2454] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3043751cae6f4656b9f93278a61a08ae in term 1.
I20260927 03:26:42.218606 17517 raft_consensus.cc:2454] T c6081cf2fcab43dc8712b21f5bba56a3 P 421dd83aade549fc913e87ca7629e042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3043751cae6f4656b9f93278a61a08ae in term 1.
I20260927 03:26:42.218693 17585 leader_election.cc:302] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3043751cae6f4656b9f93278a61a08ae, 9dc9aaa73e244124b3cb7130cc2f9e5f; no voters:
I20260927 03:26:42.218852 17856 raft_consensus.cc:2781] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:42.219017 17856 raft_consensus.cc:686] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [term 1 LEADER]: Becoming Leader. State: Replica: 3043751cae6f4656b9f93278a61a08ae, State: Running, Role: LEADER
I20260927 03:26:42.219113 17856 consensus_queue.cc:237] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421dd83aade549fc913e87ca7629e042" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } } peers { permanent_uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36273 } } peers { permanent_uuid: "3043751cae6f4656b9f93278a61a08ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } }
I20260927 03:26:42.219930 17371 catalog_manager.cc:5495] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 3043751cae6f4656b9f93278a61a08ae (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3043751cae6f4656b9f93278a61a08ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421dd83aade549fc913e87ca7629e042" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3043751cae6f4656b9f93278a61a08ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43699 } health_report { overall_health: HEALTHY } } }
W20260927 03:26:42.221069 17564 tablet.cc:2369] T c6081cf2fcab43dc8712b21f5bba56a3 P 421dd83aade549fc913e87ca7629e042: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:26:42.228551 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.226155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45648 (local address 127.0.98.2:43699)
0927 03:26:42.226201 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.226203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.226210 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:26:42.226243 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.226244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.226263 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.226321 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:26:42.226449 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.227024 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.227798 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.228003 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.228058 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.228210 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:26:42.228328 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:26:42.228399 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.228436 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.228457 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:26:42.230376 17517 raft_consensus.cc:1261] T c6081cf2fcab43dc8712b21f5bba56a3 P 421dd83aade549fc913e87ca7629e042 [term 1 FOLLOWER]: Refusing update from remote peer 3043751cae6f4656b9f93278a61a08ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:26:42.230382 17785 raft_consensus.cc:1261] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f [term 1 FOLLOWER]: Refusing update from remote peer 3043751cae6f4656b9f93278a61a08ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:26:42.230799 17856 consensus_queue.cc:1035] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "421dd83aade549fc913e87ca7629e042" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:26:42.230975 17858 consensus_queue.cc:1035] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dc9aaa73e244124b3cb7130cc2f9e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:26:42.233853 17861 mvcc.cc:205] Tried to move back new op lower bound from 7333804450733039616 to 7333804450690195456. Current Snapshot: MvccSnapshot[applied={T|T < 7333804450733039616}]
I20260927 03:26:42.234900 17862 mvcc.cc:205] Tried to move back new op lower bound from 7333804450733039616 to 7333804450690195456. Current Snapshot: MvccSnapshot[applied={T|T < 7333804450733039616 or (T in {7333804450733039616})}]
I20260927 03:26:42.235008 17864 mvcc.cc:205] Tried to move back new op lower bound from 7333804450733039616 to 7333804450690195456. Current Snapshot: MvccSnapshot[applied={T|T < 7333804450733039616 or (T in {7333804450733039616})}]
I20260927 03:26:42.236254 17371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:26:42.236374 17371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:26:42.237284 17371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ca9268473de4453b880cf7fcd5a896f: Sending DeleteTablet for 3 replicas of tablet c6081cf2fcab43dc8712b21f5bba56a3
I20260927 03:26:42.237623 17765 tablet_service.cc:1514] Processing DeleteTablet for tablet c6081cf2fcab43dc8712b21f5bba56a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56110
I20260927 03:26:42.237639 17497 tablet_service.cc:1514] Processing DeleteTablet for tablet c6081cf2fcab43dc8712b21f5bba56a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35896
I20260927 03:26:42.237687 17631 tablet_service.cc:1514] Processing DeleteTablet for tablet c6081cf2fcab43dc8712b21f5bba56a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45638
I20260927 03:26:42.237929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 17434
I20260927 03:26:42.238011 17871 tablet_replica.cc:330] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f: stopping tablet replica
I20260927 03:26:42.238103 17872 tablet_replica.cc:330] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae: stopping tablet replica
I20260927 03:26:42.238196 17871 raft_consensus.cc:2229] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:26:42.238231 17872 raft_consensus.cc:2229] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:26:42.238325 17871 raft_consensus.cc:2258] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:42.238332 17872 raft_consensus.cc:2258] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:42.238888 17871 ts_tablet_manager.cc:1905] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:26:42.239015 17872 ts_tablet_manager.cc:1905] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:26:42.240844 17872 ts_tablet_manager.cc:1918] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:26:42.240855 17871 ts_tablet_manager.cc:1918] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:26:42.240921 17872 log.cc:1196] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/c6081cf2fcab43dc8712b21f5bba56a3
I20260927 03:26:42.240947 17871 log.cc:1196] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/c6081cf2fcab43dc8712b21f5bba56a3
I20260927 03:26:42.241223 17872 ts_tablet_manager.cc:1939] T c6081cf2fcab43dc8712b21f5bba56a3 P 3043751cae6f4656b9f93278a61a08ae: Deleting consensus metadata
I20260927 03:26:42.241235 17871 ts_tablet_manager.cc:1939] T c6081cf2fcab43dc8712b21f5bba56a3 P 9dc9aaa73e244124b3cb7130cc2f9e5f: Deleting consensus metadata
I20260927 03:26:42.241698 17356 catalog_manager.cc:4841] TS 3043751cae6f4656b9f93278a61a08ae (127.0.98.2:43699): tablet c6081cf2fcab43dc8712b21f5bba56a3 (table test-table [id=de697610ad9e47cea3a3b6e3f5732118]) successfully deleted
I20260927 03:26:42.241828 17357 catalog_manager.cc:4841] TS 9dc9aaa73e244124b3cb7130cc2f9e5f (127.0.98.3:36273): tablet c6081cf2fcab43dc8712b21f5bba56a3 (table test-table [id=de697610ad9e47cea3a3b6e3f5732118]) successfully deleted
W20260927 03:26:42.244640 17356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40537 (error 108)
I20260927 03:26:42.244799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 17568
I20260927 03:26:42.245738 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.245620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40537 (local address 127.0.0.1:35900)
0927 03:26:42.245671 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:42.245708 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 03:26:42.245853 17356 catalog_manager.cc:4584] TS 421dd83aade549fc913e87ca7629e042 (127.0.98.1:40537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6081cf2fcab43dc8712b21f5bba56a3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40537: connect: Connection refused (error 111)
I20260927 03:26:42.249686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 17702
I20260927 03:26:42.253441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 17340
2026-09-27T03:26:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1940 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T03:26:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:26:42Z Disabled control of system clock
I20260927 03:26:42.279381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44693
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:26:42.359194 17877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:42.359436 17877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:42.359457 17877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:42.360633 17877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:42.360669 17877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:42.360680 17877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:42.360693 17877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:42.360703 17877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:42.362028 17877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44693
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:42.362229 17877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:42.362475 17877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:42.367494 17885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:42.367507 17883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:42.367684 17877 server_base.cc:1034] running on GCE node
W20260927 03:26:42.367507 17882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:42.368043 17877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:42.368405 17877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:42.369585 17877 hybrid_clock.cc:648] HybridClock initialized: now 1790479602369525 us; error 84 us; skew 500 ppm
I20260927 03:26:42.376681 17877 webserver.cc:460] Webserver started at http://127.0.98.62:45717/ using document root <none> and password file <none>
I20260927 03:26:42.376966 17877 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:42.377022 17877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:42.377163 17877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:42.378271 17877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "5fa0ae6e3bdf42fda35414179b4ba07d"
format_stamp: "Formatted at 2026-09-27 03:26:42 on dist-test-slave-gw2v"
server_key: "f68b3d968c54c739ae071a1bcfe28ba7"
server_key_iv: "3821cb82882884a63e0f67be12820789"
server_key_version: "encryptionkey@0"
I20260927 03:26:42.378599 17877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "5fa0ae6e3bdf42fda35414179b4ba07d"
format_stamp: "Formatted at 2026-09-27 03:26:42 on dist-test-slave-gw2v"
server_key: "f68b3d968c54c739ae071a1bcfe28ba7"
server_key_iv: "3821cb82882884a63e0f67be12820789"
server_key_version: "encryptionkey@0"
I20260927 03:26:42.379998 17877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:26:42.380852 17891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:42.381043 17877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:42.381131 17877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "5fa0ae6e3bdf42fda35414179b4ba07d"
format_stamp: "Formatted at 2026-09-27 03:26:42 on dist-test-slave-gw2v"
server_key: "f68b3d968c54c739ae071a1bcfe28ba7"
server_key_iv: "3821cb82882884a63e0f67be12820789"
server_key_version: "encryptionkey@0"
I20260927 03:26:42.381228 17877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:42.406687 17877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:42.407055 17877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:42.407195 17877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:42.410838 17877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44693
I20260927 03:26:42.410883 17943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44693 every 8 connection(s)
I20260927 03:26:42.411207 17877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:42.412132 17944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:42.413659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 17877
I20260927 03:26:42.413770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:26:42.413932 392 external_mini_cluster.cc:1428] Setting key dca117bca67eed13842d3031e5c8a18d
I20260927 03:26:42.414686 17944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d: Bootstrap starting.
I20260927 03:26:42.415388 17944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:42.415721 17944 log.cc:824] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:42.416430 17944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d: No bootstrap required, opened a new log
I20260927 03:26:42.418531 17944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa0ae6e3bdf42fda35414179b4ba07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44693 } }
I20260927 03:26:42.418720 17944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:42.418756 17944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa0ae6e3bdf42fda35414179b4ba07d, State: Initialized, Role: FOLLOWER
I20260927 03:26:42.418902 17944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa0ae6e3bdf42fda35414179b4ba07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44693 } }
I20260927 03:26:42.418982 17944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:42.419003 17944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:42.419029 17944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:42.419554 17944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa0ae6e3bdf42fda35414179b4ba07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44693 } }
I20260927 03:26:42.419648 17944 leader_election.cc:302] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fa0ae6e3bdf42fda35414179b4ba07d; no voters:
I20260927 03:26:42.419799 17944 leader_election.cc:288] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:26:42.419829 17949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:42.420109 17949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [term 1 LEADER]: Becoming Leader. State: Replica: 5fa0ae6e3bdf42fda35414179b4ba07d, State: Running, Role: LEADER
I20260927 03:26:42.420145 17944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:42.420120 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.414460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.0.98.62:44693)
0927 03:26:42.414665 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.414674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.414708 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:26:42.415273 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.415283 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.415365 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.415523 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:42.415820 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.417122 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.417907 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.418554 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.418682 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.419418 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:42.419422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:42.419422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:42.419619 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:42.419664 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:42.419681 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:42.419743 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.419813 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.419895 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260927 03:26:42.420212 17949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa0ae6e3bdf42fda35414179b4ba07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44693 } }
I20260927 03:26:42.420514 17949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fa0ae6e3bdf42fda35414179b4ba07d. Latest consensus state: current_term: 1 leader_uuid: "5fa0ae6e3bdf42fda35414179b4ba07d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa0ae6e3bdf42fda35414179b4ba07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44693 } } }
I20260927 03:26:42.420570 17950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fa0ae6e3bdf42fda35414179b4ba07d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa0ae6e3bdf42fda35414179b4ba07d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44693 } } }
I20260927 03:26:42.420697 17949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:42.420748 17950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:42.420997 17958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:42.421576 17958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:42.423218 17958 catalog_manager.cc:1347] Generated new cluster ID: d17b65298e1c4d5a870b998b4001166c
I20260927 03:26:42.423272 17958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:26:42.443284 17958 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:26:42.444038 17958 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:42.449394 17958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d: Generated new TSK 0
I20260927 03:26:42.449640 17958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:26:42.457302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44693
--builtin_ntp_servers=127.0.98.20:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:26:42.539081 17968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:42.539304 17968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:42.539332 17968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:42.540503 17968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:42.540546 17968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:42.540580 17968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:26:42.542016 17968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:42.542263 17968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:42.542558 17968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:42.547549 17976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:42.547570 17974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:42.547570 17973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:42.548547 17968 server_base.cc:1034] running on GCE node
I20260927 03:26:42.548750 17968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:42.549005 17968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:42.550194 17968 hybrid_clock.cc:648] HybridClock initialized: now 1790479602550171 us; error 33 us; skew 500 ppm
I20260927 03:26:42.551661 17968 webserver.cc:460] Webserver started at http://127.0.98.1:41549/ using document root <none> and password file <none>
I20260927 03:26:42.551910 17968 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:42.551954 17968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:42.552074 17968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:42.553162 17968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "5eccab052ca048189acc312482fe0958"
format_stamp: "Formatted at 2026-09-27 03:26:42 on dist-test-slave-gw2v"
server_key: "7ff4be221dce59c26d4e88424416c5c9"
server_key_iv: "dd6d6ef18b1237df2c025ef1fe6cd9a0"
server_key_version: "encryptionkey@0"
I20260927 03:26:42.553490 17968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5eccab052ca048189acc312482fe0958"
format_stamp: "Formatted at 2026-09-27 03:26:42 on dist-test-slave-gw2v"
server_key: "7ff4be221dce59c26d4e88424416c5c9"
server_key_iv: "dd6d6ef18b1237df2c025ef1fe6cd9a0"
server_key_version: "encryptionkey@0"
I20260927 03:26:42.554735 17968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:26:42.555467 17982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:42.555614 17968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:26:42.555689 17968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "5eccab052ca048189acc312482fe0958"
format_stamp: "Formatted at 2026-09-27 03:26:42 on dist-test-slave-gw2v"
server_key: "7ff4be221dce59c26d4e88424416c5c9"
server_key_iv: "dd6d6ef18b1237df2c025ef1fe6cd9a0"
server_key_version: "encryptionkey@0"
I20260927 03:26:42.555758 17968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:42.569156 17968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:42.569443 17968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:42.569562 17968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:42.569784 17968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:42.570093 17968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:42.570125 17968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:42.570147 17968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:42.570158 17968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:42.575891 17968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36281
I20260927 03:26:42.575928 18095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36281 every 8 connection(s)
I20260927 03:26:42.576287 17968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:26:42.581660 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.577097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49367 (local address 127.0.98.62:44693)
0927 03:26:42.577149 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.577151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.577451 (+ 300us) server_negotiation.cc:407] Connection header received
0927 03:26:42.578023 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.578024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.578042 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.578094 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:42.578752 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.579286 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.580288 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.580485 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.580547 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.581353 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:42.581354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:42.581354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:42.581358 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:42.581372 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:42.581375 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:42.581415 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.581519 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.581539 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:26:42.581847 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.577100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44693 (local address 127.0.98.1:49367)
0927 03:26:42.577305 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:42.577359 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:42.577928 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:42.578158 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:42.578162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:42.578170 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:42.578671 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.578676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.579353 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.579353 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.580201 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.580204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.580620 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.580621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.580734 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.581220 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:26:42.581221 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:42.581252 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:26:42.581277 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:42.581447 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:42.581451 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:42.581454 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:42.581544 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 03:26:42.581626 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260927 03:26:42.582229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 17968
I20260927 03:26:42.582336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:26:42.582337 18096 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44693
I20260927 03:26:42.582424 18096 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:42.582597 392 external_mini_cluster.cc:1428] Setting key 55de940837e473e84764a2686e3cefe3
I20260927 03:26:42.582671 18096 heartbeater.cc:511] Master 127.0.98.62:44693 requested a full tablet report, sending...
I20260927 03:26:42.583226 17907 ts_manager.cc:194] Registered new tserver with Master: 5eccab052ca048189acc312482fe0958 (127.0.98.1:36281)
I20260927 03:26:42.583514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44693
--builtin_ntp_servers=127.0.98.20:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:26:42.583882 17907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49367
W20260927 03:26:42.663086 18099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:42.663302 18099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:42.663323 18099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:42.664479 18099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:42.664512 18099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:42.664536 18099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:26:42.665853 18099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:42.666054 18099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:42.666318 18099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:42.671271 18105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:42.671355 18099 server_base.cc:1034] running on GCE node
W20260927 03:26:42.671279 18104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:42.671572 18107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:42.671823 18099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:42.672066 18099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:42.673214 18099 hybrid_clock.cc:648] HybridClock initialized: now 1790479602673201 us; error 28 us; skew 500 ppm
I20260927 03:26:42.674559 18099 webserver.cc:460] Webserver started at http://127.0.98.2:44013/ using document root <none> and password file <none>
I20260927 03:26:42.674845 18099 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:42.674906 18099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:42.675040 18099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:42.676134 18099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "5428f36a570f405f8ee8e61f618e7bff"
format_stamp: "Formatted at 2026-09-27 03:26:42 on dist-test-slave-gw2v"
server_key: "85dd925bf63403c51971bc4ca9761d8c"
server_key_iv: "d791a7c44ea9a63fe23a02c76d4abbec"
server_key_version: "encryptionkey@0"
I20260927 03:26:42.676481 18099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5428f36a570f405f8ee8e61f618e7bff"
format_stamp: "Formatted at 2026-09-27 03:26:42 on dist-test-slave-gw2v"
server_key: "85dd925bf63403c51971bc4ca9761d8c"
server_key_iv: "d791a7c44ea9a63fe23a02c76d4abbec"
server_key_version: "encryptionkey@0"
I20260927 03:26:42.677781 18099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:26:42.678520 18113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:42.678704 18099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:26:42.678783 18099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "5428f36a570f405f8ee8e61f618e7bff"
format_stamp: "Formatted at 2026-09-27 03:26:42 on dist-test-slave-gw2v"
server_key: "85dd925bf63403c51971bc4ca9761d8c"
server_key_iv: "d791a7c44ea9a63fe23a02c76d4abbec"
server_key_version: "encryptionkey@0"
I20260927 03:26:42.678879 18099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:42.694764 18099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:42.695147 18099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:42.695279 18099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:42.695494 18099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:42.695820 18099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:42.695854 18099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:42.695883 18099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:42.695906 18099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:42.701210 18099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32789
I20260927 03:26:42.701263 18226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32789 every 8 connection(s)
I20260927 03:26:42.701583 18099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:26:42.706862 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.702305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44457 (local address 127.0.98.62:44693)
0927 03:26:42.702370 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.702372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.702684 (+ 312us) server_negotiation.cc:407] Connection header received
0927 03:26:42.703232 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.703233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.703246 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.703293 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:42.704099 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.704589 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.705533 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.705725 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.705781 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.706577 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:42.706578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:42.706578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:42.706583 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:42.706601 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:42.706605 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:42.706643 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.706717 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.706744 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:26:42.707067 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.702275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44693 (local address 127.0.98.2:44457)
0927 03:26:42.702476 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:42.702512 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:42.703133 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:42.703403 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:42.703407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:42.703415 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:42.703987 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.703992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.704674 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.704676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.705449 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.705452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.705797 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.705798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.705920 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.706443 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:26:42.706444 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:42.706473 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:26:42.706495 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:42.706645 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:42.706651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:42.706654 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:42.706714 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:26:42.706849 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260927 03:26:42.707515 18227 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44693
I20260927 03:26:42.707602 18227 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:42.707762 18227 heartbeater.cc:511] Master 127.0.98.62:44693 requested a full tablet report, sending...
I20260927 03:26:42.708197 17907 ts_manager.cc:194] Registered new tserver with Master: 5428f36a570f405f8ee8e61f618e7bff (127.0.98.2:32789)
I20260927 03:26:42.708256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 18099
I20260927 03:26:42.708350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:26:42.708531 392 external_mini_cluster.cc:1428] Setting key aff7b871dc1e29ef335b9666835c37a6
I20260927 03:26:42.708691 17907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44457
I20260927 03:26:42.709522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44693
--builtin_ntp_servers=127.0.98.20:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:26:42.790002 18230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:42.790230 18230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:42.790251 18230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:42.791432 18230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:42.791473 18230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:42.791496 18230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:26:42.792793 18230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:42.792982 18230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:42.793241 18230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:42.798106 18235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:42.798133 18230 server_base.cc:1034] running on GCE node
W20260927 03:26:42.798136 18236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:42.798106 18238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:42.798563 18230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:42.798791 18230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:42.799949 18230 hybrid_clock.cc:648] HybridClock initialized: now 1790479602799940 us; error 27 us; skew 500 ppm
I20260927 03:26:42.801240 18230 webserver.cc:460] Webserver started at http://127.0.98.3:36331/ using document root <none> and password file <none>
I20260927 03:26:42.801460 18230 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:42.801509 18230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:42.801633 18230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:42.802693 18230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "82e7783fadd44d6f9489a6cc04557bab"
format_stamp: "Formatted at 2026-09-27 03:26:42 on dist-test-slave-gw2v"
server_key: "7ff106da5bbc2cba42e19aea70b6aeac"
server_key_iv: "8a8dfb137e1d4170685f2a199deb3a93"
server_key_version: "encryptionkey@0"
I20260927 03:26:42.803046 18230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "82e7783fadd44d6f9489a6cc04557bab"
format_stamp: "Formatted at 2026-09-27 03:26:42 on dist-test-slave-gw2v"
server_key: "7ff106da5bbc2cba42e19aea70b6aeac"
server_key_iv: "8a8dfb137e1d4170685f2a199deb3a93"
server_key_version: "encryptionkey@0"
I20260927 03:26:42.804236 18230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:42.804896 18244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:42.805035 18230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:26:42.805114 18230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "82e7783fadd44d6f9489a6cc04557bab"
format_stamp: "Formatted at 2026-09-27 03:26:42 on dist-test-slave-gw2v"
server_key: "7ff106da5bbc2cba42e19aea70b6aeac"
server_key_iv: "8a8dfb137e1d4170685f2a199deb3a93"
server_key_version: "encryptionkey@0"
I20260927 03:26:42.805207 18230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:42.818296 18230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:42.818604 18230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:42.818725 18230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:42.818965 18230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:42.819300 18230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:42.819334 18230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:42.819362 18230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:42.819387 18230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:42.824970 18230 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38739
I20260927 03:26:42.825024 18357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38739 every 8 connection(s)
I20260927 03:26:42.825343 18230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:26:42.830562 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.825984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41347 (local address 127.0.98.62:44693)
0927 03:26:42.826069 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.826071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.826264 (+ 193us) server_negotiation.cc:407] Connection header received
0927 03:26:42.826929 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.826931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.826942 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.826990 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:42.827697 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.828218 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.829206 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.829395 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.829447 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.830267 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:42.830269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:42.830269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:42.830275 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:42.830300 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:42.830304 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:42.830365 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.830469 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.830492 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 03:26:42.830736 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.825970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44693 (local address 127.0.98.3:41347)
0927 03:26:42.826148 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:42.826181 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:42.826783 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:42.827053 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:42.827056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:42.827065 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:42.827561 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.827567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.828316 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.828317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.829117 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.829120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.829467 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.829468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.829586 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.830106 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:26:42.830107 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:42.830136 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:26:42.830163 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:42.830370 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:42.830373 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:42.830375 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:26:42.830425 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:26:42.830511 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260927 03:26:42.831274 18358 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44693
I20260927 03:26:42.831364 18358 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:42.831542 18358 heartbeater.cc:511] Master 127.0.98.62:44693 requested a full tablet report, sending...
I20260927 03:26:42.831928 17907 ts_manager.cc:194] Registered new tserver with Master: 82e7783fadd44d6f9489a6cc04557bab (127.0.98.3:38739)
I20260927 03:26:42.832280 17907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:41347
I20260927 03:26:42.834129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 18230
I20260927 03:26:42.834226 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:26:42.834404 392 external_mini_cluster.cc:1428] Setting key 55db2cf07196069068cbb0c05a9c8486
I20260927 03:26:42.835198 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:26:42.839169 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.836440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.0.98.62:44693)
0927 03:26:42.836488 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.836489 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.836603 (+ 114us) server_negotiation.cc:407] Connection header received
0927 03:26:42.836641 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.836641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.836651 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.836683 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:42.836798 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.837267 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.837958 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.838153 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.838204 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.838789 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:42.838790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:42.838791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:42.838795 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:42.838808 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:42.838812 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:42.838889 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.839052 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.839087 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:26:42.840531 17907 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:26:42.842044 17907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:26:42.848340 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.844184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32789 (local address 127.0.0.1:48808)
0927 03:26:42.844410 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:42.844451 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:42.844521 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:42.844997 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:42.844999 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:42.845001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:42.845050 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.845052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.846357 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.846358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.847390 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.847392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.848131 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.848132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.848185 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.848188 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:42.848228 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 03:26:42.848257 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":26,"thread_start_us":102,"threads_started":1}
I20260927 03:26:42.848472 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.844184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38739 (local address 127.0.0.1:41178)
0927 03:26:42.844410 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:42.844451 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:42.844516 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:42.845186 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:42.845186 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:42.845188 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:42.845220 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.845221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.846418 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.846419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.847349 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.847352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.848201 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.848202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.848285 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.848289 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:26:42.848344 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:26:42.848384 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
I20260927 03:26:42.848476 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.844489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.0.98.3:38739)
0927 03:26:42.845049 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.845054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.845063 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:26:42.845099 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.845102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.845117 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.845189 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:42.845268 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.846323 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.847438 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.848124 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.848185 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.848273 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.848358 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.848394 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":74,"threads_started":1}
I20260927 03:26:42.848680 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.844555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48808 (local address 127.0.98.2:32789)
0927 03:26:42.844712 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.844717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.844728 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:26:42.844846 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.844849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.844864 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.844928 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:42.845128 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.846270 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.847461 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.847948 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.848011 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.848101 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.848544 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.848586 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260927 03:26:42.848716 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.844375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.0.98.1:36281)
0927 03:26:42.844765 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:42.844769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:42.844777 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:26:42.844815 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:42.844818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:42.844833 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:42.844921 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:42.845121 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.845888 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.847209 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.847927 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.848001 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.848097 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:42.848583 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 03:26:42.848626 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":83,"threads_started":1}
I20260927 03:26:42.848340 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:42.844184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36281 (local address 127.0.0.1:57228)
0927 03:26:42.844629 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:42.844670 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:42.844725 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:42.844974 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:42.844977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:42.844980 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:42.845049 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.845051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.845960 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.845961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.847124 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:42.847127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:42.848012 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:42.848013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:42.848087 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:42.848100 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 03:26:42.848182 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 03:26:42.848239 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":21,"thread_start_us":96,"threads_started":1}
I20260927 03:26:42.849220 18292 tablet_service.cc:1467] Processing CreateTablet for tablet 09d0881806424c5c981b02e447c09632 (DEFAULT_TABLE table=test-table [id=a9d582c452af465e8a3aaac18cf8a02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:42.849277 18161 tablet_service.cc:1467] Processing CreateTablet for tablet 09d0881806424c5c981b02e447c09632 (DEFAULT_TABLE table=test-table [id=a9d582c452af465e8a3aaac18cf8a02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:42.849594 18292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d0881806424c5c981b02e447c09632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:42.849594 18161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d0881806424c5c981b02e447c09632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:42.849623 18030 tablet_service.cc:1467] Processing CreateTablet for tablet 09d0881806424c5c981b02e447c09632 (DEFAULT_TABLE table=test-table [id=a9d582c452af465e8a3aaac18cf8a02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:42.849951 18030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d0881806424c5c981b02e447c09632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:42.852452 18372 tablet_bootstrap.cc:492] T 09d0881806424c5c981b02e447c09632 P 5eccab052ca048189acc312482fe0958: Bootstrap starting.
I20260927 03:26:42.853062 18374 tablet_bootstrap.cc:492] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab: Bootstrap starting.
I20260927 03:26:42.853207 18372 tablet_bootstrap.cc:654] T 09d0881806424c5c981b02e447c09632 P 5eccab052ca048189acc312482fe0958: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:42.853511 18372 log.cc:824] T 09d0881806424c5c981b02e447c09632 P 5eccab052ca048189acc312482fe0958: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:42.853688 18374 tablet_bootstrap.cc:654] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:42.853940 18374 log.cc:824] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:42.854192 18373 tablet_bootstrap.cc:492] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff: Bootstrap starting.
I20260927 03:26:42.854269 18372 tablet_bootstrap.cc:492] T 09d0881806424c5c981b02e447c09632 P 5eccab052ca048189acc312482fe0958: No bootstrap required, opened a new log
I20260927 03:26:42.854331 18372 ts_tablet_manager.cc:1397] T 09d0881806424c5c981b02e447c09632 P 5eccab052ca048189acc312482fe0958: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:26:42.854581 18374 tablet_bootstrap.cc:492] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab: No bootstrap required, opened a new log
I20260927 03:26:42.854640 18374 ts_tablet_manager.cc:1397] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:26:42.854857 18373 tablet_bootstrap.cc:654] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:42.855178 18373 log.cc:824] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:42.855899 18373 tablet_bootstrap.cc:492] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff: No bootstrap required, opened a new log
I20260927 03:26:42.855966 18373 ts_tablet_manager.cc:1397] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:26:42.856781 18374 raft_consensus.cc:348] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5428f36a570f405f8ee8e61f618e7bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32789 } } peers { permanent_uuid: "5eccab052ca048189acc312482fe0958" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36281 } } peers { permanent_uuid: "82e7783fadd44d6f9489a6cc04557bab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260927 03:26:42.856784 18372 raft_consensus.cc:348] T 09d0881806424c5c981b02e447c09632 P 5eccab052ca048189acc312482fe0958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5428f36a570f405f8ee8e61f618e7bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32789 } } peers { permanent_uuid: "5eccab052ca048189acc312482fe0958" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36281 } } peers { permanent_uuid: "82e7783fadd44d6f9489a6cc04557bab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260927 03:26:42.856912 18374 raft_consensus.cc:374] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:42.856940 18372 raft_consensus.cc:374] T 09d0881806424c5c981b02e447c09632 P 5eccab052ca048189acc312482fe0958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:42.856945 18374 raft_consensus.cc:729] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e7783fadd44d6f9489a6cc04557bab, State: Initialized, Role: FOLLOWER
I20260927 03:26:42.856977 18372 raft_consensus.cc:729] T 09d0881806424c5c981b02e447c09632 P 5eccab052ca048189acc312482fe0958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eccab052ca048189acc312482fe0958, State: Initialized, Role: FOLLOWER
I20260927 03:26:42.857080 18374 consensus_queue.cc:260] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5428f36a570f405f8ee8e61f618e7bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32789 } } peers { permanent_uuid: "5eccab052ca048189acc312482fe0958" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36281 } } peers { permanent_uuid: "82e7783fadd44d6f9489a6cc04557bab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260927 03:26:42.857102 18372 consensus_queue.cc:260] T 09d0881806424c5c981b02e447c09632 P 5eccab052ca048189acc312482fe0958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5428f36a570f405f8ee8e61f618e7bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32789 } } peers { permanent_uuid: "5eccab052ca048189acc312482fe0958" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36281 } } peers { permanent_uuid: "82e7783fadd44d6f9489a6cc04557bab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260927 03:26:42.857335 18374 ts_tablet_manager.cc:1428] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:26:42.857352 18372 ts_tablet_manager.cc:1428] T 09d0881806424c5c981b02e447c09632 P 5eccab052ca048189acc312482fe0958: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:26:42.857448 18358 heartbeater.cc:503] Master 127.0.98.62:44693 was elected leader, sending a full tablet report...
I20260927 03:26:42.857455 18096 heartbeater.cc:503] Master 127.0.98.62:44693 was elected leader, sending a full tablet report...
I20260927 03:26:42.858294 18373 raft_consensus.cc:348] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5428f36a570f405f8ee8e61f618e7bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32789 } } peers { permanent_uuid: "5eccab052ca048189acc312482fe0958" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36281 } } peers { permanent_uuid: "82e7783fadd44d6f9489a6cc04557bab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260927 03:26:42.858440 18373 raft_consensus.cc:374] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:42.858479 18373 raft_consensus.cc:729] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5428f36a570f405f8ee8e61f618e7bff, State: Initialized, Role: FOLLOWER
I20260927 03:26:42.858618 18373 consensus_queue.cc:260] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5428f36a570f405f8ee8e61f618e7bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32789 } } peers { permanent_uuid: "5eccab052ca048189acc312482fe0958" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36281 } } peers { permanent_uuid: "82e7783fadd44d6f9489a6cc04557bab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260927 03:26:42.858897 18373 ts_tablet_manager.cc:1428] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:26:42.858920 18227 heartbeater.cc:503] Master 127.0.98.62:44693 was elected leader, sending a full tablet report...
W20260927 03:26:42.952134 18228 tablet.cc:2369] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:26:43.075855 18359 tablet.cc:2369] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:26:43.077085 18097 tablet.cc:2369] T 09d0881806424c5c981b02e447c09632 P 5eccab052ca048189acc312482fe0958: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:26:43.163744 18380 raft_consensus.cc:482] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:26:43.163868 18380 raft_consensus.cc:504] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5428f36a570f405f8ee8e61f618e7bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32789 } } peers { permanent_uuid: "5eccab052ca048189acc312482fe0958" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36281 } } peers { permanent_uuid: "82e7783fadd44d6f9489a6cc04557bab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260927 03:26:43.164244 18380 leader_election.cc:288] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5eccab052ca048189acc312482fe0958 (127.0.98.1:36281), 82e7783fadd44d6f9489a6cc04557bab (127.0.98.3:38739)
I20260927 03:26:43.168040 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:43.164578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53719 (local address 127.0.98.3:38739)
0927 03:26:43.164626 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:43.164628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:43.164705 (+ 77us) server_negotiation.cc:407] Connection header received
0927 03:26:43.164739 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:43.164739 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:43.164746 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:43.164786 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:43.164918 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:43.165604 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:43.167141 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:43.167683 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:43.167757 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:43.167768 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:43.167925 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 03:26:43.167965 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:26:43.168042 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:43.164504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46661 (local address 127.0.98.1:36281)
0927 03:26:43.164554 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:43.164555 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:43.164566 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:26:43.164596 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:43.164597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:43.164613 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:43.164667 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:43.164843 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:43.165666 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:43.167141 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:43.167691 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:43.167761 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:43.167768 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:43.167925 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 03:26:43.167965 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:26:43.168046 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:43.164424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36281 (local address 127.0.98.2:46661)
0927 03:26:43.164483 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:43.164495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:43.164545 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:43.164693 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:43.164694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:43.164696 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:43.164768 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:43.164769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:43.165735 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:43.165736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:43.167039 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:43.167041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:43.167767 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:43.167768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:43.167854 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:43.167859 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:26:43.167908 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:26:43.167952 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":64}
I20260927 03:26:43.168051 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:43.164445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38739 (local address 127.0.98.2:53719)
0927 03:26:43.164632 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:43.164640 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:43.164698 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:43.164807 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:43.164808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:43.164809 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:43.164863 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:43.164864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:43.165680 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:43.165681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:43.167037 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:43.167041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:43.167767 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:43.167768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:43.167854 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:43.167859 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:26:43.167908 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:26:43.167951 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":59,"thread_start_us":96,"threads_started":1}
I20260927 03:26:43.168437 18050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d0881806424c5c981b02e447c09632" candidate_uuid: "5428f36a570f405f8ee8e61f618e7bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eccab052ca048189acc312482fe0958" is_pre_election: true
I20260927 03:26:43.168460 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d0881806424c5c981b02e447c09632" candidate_uuid: "5428f36a570f405f8ee8e61f618e7bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82e7783fadd44d6f9489a6cc04557bab" is_pre_election: true
I20260927 03:26:43.168591 18050 raft_consensus.cc:2454] T 09d0881806424c5c981b02e447c09632 P 5eccab052ca048189acc312482fe0958 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5428f36a570f405f8ee8e61f618e7bff in term 0.
I20260927 03:26:43.168606 18312 raft_consensus.cc:2454] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5428f36a570f405f8ee8e61f618e7bff in term 0.
I20260927 03:26:43.168825 18117 leader_election.cc:302] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5428f36a570f405f8ee8e61f618e7bff, 5eccab052ca048189acc312482fe0958, 82e7783fadd44d6f9489a6cc04557bab; no voters:
I20260927 03:26:43.168993 18380 raft_consensus.cc:2781] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:26:43.169056 18380 raft_consensus.cc:482] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:26:43.169071 18380 raft_consensus.cc:3037] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:43.169708 18380 raft_consensus.cc:504] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5428f36a570f405f8ee8e61f618e7bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32789 } } peers { permanent_uuid: "5eccab052ca048189acc312482fe0958" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36281 } } peers { permanent_uuid: "82e7783fadd44d6f9489a6cc04557bab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260927 03:26:43.169832 18380 leader_election.cc:288] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [CANDIDATE]: Term 1 election: Requested vote from peers 5eccab052ca048189acc312482fe0958 (127.0.98.1:36281), 82e7783fadd44d6f9489a6cc04557bab (127.0.98.3:38739)
I20260927 03:26:43.169977 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d0881806424c5c981b02e447c09632" candidate_uuid: "5428f36a570f405f8ee8e61f618e7bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82e7783fadd44d6f9489a6cc04557bab"
I20260927 03:26:43.169983 18050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d0881806424c5c981b02e447c09632" candidate_uuid: "5428f36a570f405f8ee8e61f618e7bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eccab052ca048189acc312482fe0958"
I20260927 03:26:43.170049 18312 raft_consensus.cc:3037] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:43.170054 18050 raft_consensus.cc:3037] T 09d0881806424c5c981b02e447c09632 P 5eccab052ca048189acc312482fe0958 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:43.170765 18312 raft_consensus.cc:2454] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5428f36a570f405f8ee8e61f618e7bff in term 1.
I20260927 03:26:43.170953 18116 leader_election.cc:302] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5428f36a570f405f8ee8e61f618e7bff, 82e7783fadd44d6f9489a6cc04557bab; no voters:
I20260927 03:26:43.171036 18380 raft_consensus.cc:2781] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:43.171108 18050 raft_consensus.cc:2454] T 09d0881806424c5c981b02e447c09632 P 5eccab052ca048189acc312482fe0958 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5428f36a570f405f8ee8e61f618e7bff in term 1.
I20260927 03:26:43.171165 18380 raft_consensus.cc:686] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [term 1 LEADER]: Becoming Leader. State: Replica: 5428f36a570f405f8ee8e61f618e7bff, State: Running, Role: LEADER
I20260927 03:26:43.171262 18380 consensus_queue.cc:237] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5428f36a570f405f8ee8e61f618e7bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32789 } } peers { permanent_uuid: "5eccab052ca048189acc312482fe0958" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36281 } } peers { permanent_uuid: "82e7783fadd44d6f9489a6cc04557bab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260927 03:26:43.172289 17908 catalog_manager.cc:5495] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 5428f36a570f405f8ee8e61f618e7bff (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5428f36a570f405f8ee8e61f618e7bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5428f36a570f405f8ee8e61f618e7bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5eccab052ca048189acc312482fe0958" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82e7783fadd44d6f9489a6cc04557bab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } health_report { overall_health: UNKNOWN } } }
I20260927 03:26:43.204828 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:43.202309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.0.98.2:32789)
0927 03:26:43.202364 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:43.202366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:43.202375 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:26:43.202435 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:43.202436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:43.202462 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:43.202550 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:26:43.202737 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:43.203248 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:43.204032 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:43.204234 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:43.204285 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:43.204408 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:26:43.204512 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:26:43.204569 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:43.204718 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 03:26:43.204739 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:26:43.206677 18312 raft_consensus.cc:1261] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab [term 1 FOLLOWER]: Refusing update from remote peer 5428f36a570f405f8ee8e61f618e7bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:26:43.206677 18050 raft_consensus.cc:1261] T 09d0881806424c5c981b02e447c09632 P 5eccab052ca048189acc312482fe0958 [term 1 FOLLOWER]: Refusing update from remote peer 5428f36a570f405f8ee8e61f618e7bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:26:43.207069 18380 consensus_queue.cc:1035] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eccab052ca048189acc312482fe0958" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:26:43.207206 18382 consensus_queue.cc:1035] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "82e7783fadd44d6f9489a6cc04557bab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:26:43.209486 18385 mvcc.cc:205] Tried to move back new op lower bound from 7333804454731694080 to 7333804454590164992. Current Snapshot: MvccSnapshot[applied={T|T < 7333804454731694080}]
I20260927 03:26:43.209794 18387 mvcc.cc:205] Tried to move back new op lower bound from 7333804454731694080 to 7333804454590164992. Current Snapshot: MvccSnapshot[applied={T|T < 7333804454731694080}]
I20260927 03:26:43.210590 18386 mvcc.cc:205] Tried to move back new op lower bound from 7333804454731694080 to 7333804454590164992. Current Snapshot: MvccSnapshot[applied={T|T < 7333804454731694080}]
I20260927 03:26:43.212051 17907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:26:43.212136 17907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:26:43.212841 17907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fa0ae6e3bdf42fda35414179b4ba07d: Sending DeleteTablet for 3 replicas of tablet 09d0881806424c5c981b02e447c09632
I20260927 03:26:43.213153 18161 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d0881806424c5c981b02e447c09632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:43 UTC) from {username='slave'} at 127.0.0.1:48808
I20260927 03:26:43.213177 18030 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d0881806424c5c981b02e447c09632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:43 UTC) from {username='slave'} at 127.0.0.1:57228
I20260927 03:26:43.213387 18292 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d0881806424c5c981b02e447c09632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:43 UTC) from {username='slave'} at 127.0.0.1:41178
I20260927 03:26:43.213423 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 17968
I20260927 03:26:43.214424 18396 tablet_replica.cc:330] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab: stopping tablet replica
I20260927 03:26:43.214531 18396 raft_consensus.cc:2229] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:26:43.214632 18396 raft_consensus.cc:2258] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:43.214779 18394 tablet_replica.cc:330] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff: stopping tablet replica
I20260927 03:26:43.214879 18394 raft_consensus.cc:2229] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:26:43.214962 18394 raft_consensus.cc:2258] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:43.215358 18396 ts_tablet_manager.cc:1905] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:26:43.215366 18394 ts_tablet_manager.cc:1905] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:26:43.216951 18396 ts_tablet_manager.cc:1918] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:26:43.217070 18396 log.cc:1196] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/09d0881806424c5c981b02e447c09632
I20260927 03:26:43.217057 18394 ts_tablet_manager.cc:1918] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:26:43.217118 18394 log.cc:1196] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/09d0881806424c5c981b02e447c09632
I20260927 03:26:43.217351 18394 ts_tablet_manager.cc:1939] T 09d0881806424c5c981b02e447c09632 P 5428f36a570f405f8ee8e61f618e7bff: Deleting consensus metadata
I20260927 03:26:43.217351 18396 ts_tablet_manager.cc:1939] T 09d0881806424c5c981b02e447c09632 P 82e7783fadd44d6f9489a6cc04557bab: Deleting consensus metadata
I20260927 03:26:43.217733 17893 catalog_manager.cc:4841] TS 5428f36a570f405f8ee8e61f618e7bff (127.0.98.2:32789): tablet 09d0881806424c5c981b02e447c09632 (table test-table [id=a9d582c452af465e8a3aaac18cf8a02a]) successfully deleted
I20260927 03:26:43.217777 17894 catalog_manager.cc:4841] TS 82e7783fadd44d6f9489a6cc04557bab (127.0.98.3:38739): tablet 09d0881806424c5c981b02e447c09632 (table test-table [id=a9d582c452af465e8a3aaac18cf8a02a]) successfully deleted
W20260927 03:26:43.219873 17895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36281 (error 108)
I20260927 03:26:43.220101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 18099
I20260927 03:26:43.220772 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:43.220248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36281 (local address 127.0.0.1:57240)
0927 03:26:43.220676 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:43.220738 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406}
W20260927 03:26:43.220914 17895 catalog_manager.cc:4584] TS 5eccab052ca048189acc312482fe0958 (127.0.98.1:36281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09d0881806424c5c981b02e447c09632: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36281: connect: Connection refused (error 111)
I20260927 03:26:43.224921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 18230
I20260927 03:26:43.228886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 17877
2026-09-27T03:26:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (974 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:26:43 dist-test-slave-gw2v krb5kdc[18400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:26:43 dist-test-slave-gw2v krb5kdc[18400](info): set up 2 sockets
Sep 27 03:26:43 dist-test-slave-gw2v krb5kdc[18400](info): commencing operation
krb5kdc: starting...
W20260927 03:26:44.178416 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:26:44 dist-test-slave-gw2v krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:26:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:26:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:26:44.294358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34141
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:44.375079 18413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:44.375303 18413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:44.375325 18413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:44.376579 18413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:44.376631 18413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:44.376645 18413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:44.376657 18413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:44.376667 18413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:44.378194 18413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:44.378434 18413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:44.378655 18413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:44.383334 18419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:44.383342 18421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:44.383448 18413 server_base.cc:1034] running on GCE node
W20260927 03:26:44.383524 18418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:44.383800 18413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:44.384073 18413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:44.385257 18413 hybrid_clock.cc:648] HybridClock initialized: now 1790479604385244 us; error 30 us; skew 500 ppm
Sep 27 03:26:44 dist-test-slave-gw2v krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:44.387739 18413 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:26:44.388152 18413 webserver.cc:460] Webserver started at http://127.0.98.62:42083/ using document root <none> and password file <none>
I20260927 03:26:44.388355 18413 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:44.388399 18413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:44.388496 18413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:44.389472 18413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "516a0c8b4f714beb9b54a6d85ca790dc"
format_stamp: "Formatted at 2026-09-27 03:26:44 on dist-test-slave-gw2v"
server_key: "14c8ec06ef5de21226b6257ef85f4525"
server_key_iv: "703b1916c30fcf0db3cd11642ef5256c"
server_key_version: "encryptionkey@0"
I20260927 03:26:44.389765 18413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "516a0c8b4f714beb9b54a6d85ca790dc"
format_stamp: "Formatted at 2026-09-27 03:26:44 on dist-test-slave-gw2v"
server_key: "14c8ec06ef5de21226b6257ef85f4525"
server_key_iv: "703b1916c30fcf0db3cd11642ef5256c"
server_key_version: "encryptionkey@0"
I20260927 03:26:44.390934 18413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:26:44.391558 18428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:44.391693 18413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:26:44.391768 18413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "516a0c8b4f714beb9b54a6d85ca790dc"
format_stamp: "Formatted at 2026-09-27 03:26:44 on dist-test-slave-gw2v"
server_key: "14c8ec06ef5de21226b6257ef85f4525"
server_key_iv: "703b1916c30fcf0db3cd11642ef5256c"
server_key_version: "encryptionkey@0"
I20260927 03:26:44.391829 18413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:44.405105 18413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:44.406998 18413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:44.407119 18413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:44.411065 18413 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34141
I20260927 03:26:44.411111 18480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34141 every 8 connection(s)
I20260927 03:26:44.411466 18413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:44.412338 18481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:44.414770 18481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc: Bootstrap starting.
I20260927 03:26:44.415570 18481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:44.415849 18481 log.cc:824] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:44.416606 18481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc: No bootstrap required, opened a new log
I20260927 03:26:44.418249 18481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a0c8b4f714beb9b54a6d85ca790dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34141 } }
I20260927 03:26:44.418403 18481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:44.418434 18481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516a0c8b4f714beb9b54a6d85ca790dc, State: Initialized, Role: FOLLOWER
I20260927 03:26:44.418588 18481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a0c8b4f714beb9b54a6d85ca790dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34141 } }
I20260927 03:26:44.418655 18481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:44.418685 18481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:44.418732 18481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:44.419169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 18413
I20260927 03:26:44.419265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:26:44.419364 18481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a0c8b4f714beb9b54a6d85ca790dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34141 } }
I20260927 03:26:44.419463 18481 leader_election.cc:302] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 516a0c8b4f714beb9b54a6d85ca790dc; no voters:
I20260927 03:26:44.419551 392 external_mini_cluster.cc:1428] Setting key 3ee2c62cc577c8380c9c0f54d2756f0f
I20260927 03:26:44.419637 18481 leader_election.cc:288] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:26:44.419816 18484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:44.419915 18481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:44.420044 18484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [term 1 LEADER]: Becoming Leader. State: Replica: 516a0c8b4f714beb9b54a6d85ca790dc, State: Running, Role: LEADER
I20260927 03:26:44.420199 18484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a0c8b4f714beb9b54a6d85ca790dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34141 } }
I20260927 03:26:44.420681 18485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "516a0c8b4f714beb9b54a6d85ca790dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a0c8b4f714beb9b54a6d85ca790dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34141 } } }
I20260927 03:26:44.420768 18485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:44.420912 18485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 516a0c8b4f714beb9b54a6d85ca790dc. Latest consensus state: current_term: 1 leader_uuid: "516a0c8b4f714beb9b54a6d85ca790dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a0c8b4f714beb9b54a6d85ca790dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34141 } } }
I20260927 03:26:44.420964 18485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:44.421531 18495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:44.422125 18495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:44.423826 18495 catalog_manager.cc:1347] Generated new cluster ID: a84b4e49dae84d7a94d30f79eabf0399
I20260927 03:26:44.423878 18495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:26:44 dist-test-slave-gw2v krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:26:44.428680 18495 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:26:44.429371 18495 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:44.431967 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:44.420214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34828 (local address 127.0.98.62:34141)
0927 03:26:44.420374 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:44.420385 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:44.420419 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:26:44.420556 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:44.420564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:44.420638 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:44.421541 (+ 903us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:44.421670 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.423038 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.423991 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.424593 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.424690 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:44.427665 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:44.427669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:44.427669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:44.427682 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:44.430206 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:44.430617 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:44.430620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:44.430621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:44.430722 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:44.430961 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:44.430962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:44.430962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:44.431229 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:44.431378 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:44.431496 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 03:26:44.431745 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260927 03:26:44.437363 18495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc: Generated new TSK 0
I20260927 03:26:44.437583 18495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:26:44.461911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34141
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:44.542145 18507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:44.542351 18507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:44.542390 18507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:44.543577 18507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:44.543619 18507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:44.543654 18507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:26:44.545118 18507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:44.545334 18507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:44.545539 18507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:44.550307 18515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:44.550416 18513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:44.550307 18512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:44.550673 18507 server_base.cc:1034] running on GCE node
I20260927 03:26:44.550916 18507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:44.551162 18507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:44.552313 18507 hybrid_clock.cc:648] HybridClock initialized: now 1790479604552298 us; error 33 us; skew 500 ppm
Sep 27 03:26:44 dist-test-slave-gw2v krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:44.554800 18507 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:26:44.555248 18507 webserver.cc:460] Webserver started at http://127.0.98.1:35563/ using document root <none> and password file <none>
I20260927 03:26:44.555429 18507 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:44.555470 18507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:44.555559 18507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:44.556630 18507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0"
format_stamp: "Formatted at 2026-09-27 03:26:44 on dist-test-slave-gw2v"
server_key: "186dc7eab6ab56ff1a76e8fb8fba42e4"
server_key_iv: "39da1dab81bbd82287009e346e910657"
server_key_version: "encryptionkey@0"
I20260927 03:26:44.556923 18507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0"
format_stamp: "Formatted at 2026-09-27 03:26:44 on dist-test-slave-gw2v"
server_key: "186dc7eab6ab56ff1a76e8fb8fba42e4"
server_key_iv: "39da1dab81bbd82287009e346e910657"
server_key_version: "encryptionkey@0"
I20260927 03:26:44.558108 18507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:26:44.558913 18522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:44.559062 18507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:44.559134 18507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0"
format_stamp: "Formatted at 2026-09-27 03:26:44 on dist-test-slave-gw2v"
server_key: "186dc7eab6ab56ff1a76e8fb8fba42e4"
server_key_iv: "39da1dab81bbd82287009e346e910657"
server_key_version: "encryptionkey@0"
I20260927 03:26:44.559202 18507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:44.582436 18507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:44.584620 18507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:44.584753 18507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:44.585011 18507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:44.585332 18507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:44.585369 18507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:44.585392 18507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:44.585403 18507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:44.591048 18507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34155
I20260927 03:26:44.591107 18635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34155 every 8 connection(s)
I20260927 03:26:44.591409 18507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:26:44.597047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 18507
I20260927 03:26:44.597155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:26:44.597409 392 external_mini_cluster.cc:1428] Setting key 3247edc09c817cd5305cc2d1a59068ce
Sep 27 03:26:44 dist-test-slave-gw2v krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:26:44.600724 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:44.592121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35639 (local address 127.0.98.62:34141)
0927 03:26:44.592167 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:44.592169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:44.592414 (+ 245us) server_negotiation.cc:407] Connection header received
0927 03:26:44.593296 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:44.593297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:44.593308 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:44.593447 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:44.594099 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.594731 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.595788 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.595998 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.596081 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:44.598057 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:44.598059 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:44.598059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:44.598066 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:44.599417 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:44.599806 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:44.599808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:44.599808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:44.599870 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:44.600086 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:44.600087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:44.600087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:44.600209 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:44.600269 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:44.600546 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 03:26:44.600638 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:26:44.600899 18638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:44.592089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34141 (local address 127.0.98.1:35639)
0927 03:26:44.592251 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:44.592293 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:44.593186 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:44.593542 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:44.593545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:44.593797 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:44.593994 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:44.593999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.594815 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.594839 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:44.595712 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:44.595715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.596081 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.596082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:44.596184 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:44.596667 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:26:44.596676 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:44.597972 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:44.599502 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:44.599509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:44.599511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:44.599737 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:44.599937 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:44.599938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:44.599938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:44.600032 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:44.600278 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:44.600286 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:44.600419 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 03:26:44.600517 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 03:26:44.600671 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260927 03:26:44.601485 18636 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34141
I20260927 03:26:44.601572 18636 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:44.601743 18636 heartbeater.cc:511] Master 127.0.98.62:34141 requested a full tablet report, sending...
I20260927 03:26:44.602231 18445 ts_manager.cc:194] Registered new tserver with Master: 02cfcc0b0ad94a6596ed63864cc6d7c0 (127.0.98.1:34155)
I20260927 03:26:44.602878 18445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35639
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:26:44.622540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34141
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:44.702670 18641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:44.702893 18641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:44.702914 18641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:44.704020 18641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:44.704054 18641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:44.704079 18641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:26:44.705494 18641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:44.705672 18641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:44.705847 18641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:44.710590 18649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:44.710574 18646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:44.710647 18641 server_base.cc:1034] running on GCE node
W20260927 03:26:44.710565 18647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:44.710979 18641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:44.711228 18641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:44.712370 18641 hybrid_clock.cc:648] HybridClock initialized: now 1790479604712358 us; error 28 us; skew 500 ppm
Sep 27 03:26:44 dist-test-slave-gw2v krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:44.714751 18641 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:26:44.715188 18641 webserver.cc:460] Webserver started at http://127.0.98.2:35573/ using document root <none> and password file <none>
I20260927 03:26:44.715373 18641 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:44.715411 18641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:44.715512 18641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:44.716506 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "a24dd6614d334e769a3f9468d36eccc3"
format_stamp: "Formatted at 2026-09-27 03:26:44 on dist-test-slave-gw2v"
server_key: "5ece837321ed4a59c4c2f96950544ea0"
server_key_iv: "bf2ed6d3d5725bd2d7edc81e5da7f62b"
server_key_version: "encryptionkey@0"
I20260927 03:26:44.716794 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a24dd6614d334e769a3f9468d36eccc3"
format_stamp: "Formatted at 2026-09-27 03:26:44 on dist-test-slave-gw2v"
server_key: "5ece837321ed4a59c4c2f96950544ea0"
server_key_iv: "bf2ed6d3d5725bd2d7edc81e5da7f62b"
server_key_version: "encryptionkey@0"
I20260927 03:26:44.718084 18641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:26:44.718780 18656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:44.718997 18641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:26:44.719077 18641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "a24dd6614d334e769a3f9468d36eccc3"
format_stamp: "Formatted at 2026-09-27 03:26:44 on dist-test-slave-gw2v"
server_key: "5ece837321ed4a59c4c2f96950544ea0"
server_key_iv: "bf2ed6d3d5725bd2d7edc81e5da7f62b"
server_key_version: "encryptionkey@0"
I20260927 03:26:44.719156 18641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:44.752739 18641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:44.754681 18641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:44.754807 18641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:44.755069 18641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:44.755378 18641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:44.755414 18641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:44.755436 18641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:44.755477 18641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:44.760859 18641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42881
I20260927 03:26:44.760932 18769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42881 every 8 connection(s)
I20260927 03:26:44.761209 18641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:26:44 dist-test-slave-gw2v krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:26:44.767619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 18641
I20260927 03:26:44.767717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:26:44.767916 392 external_mini_cluster.cc:1428] Setting key 74e4a9590bc76073eee8d3437a7e648a
I20260927 03:26:44.770677 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:44.761901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52467 (local address 127.0.98.62:34141)
0927 03:26:44.761943 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:44.761945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:44.762143 (+ 198us) server_negotiation.cc:407] Connection header received
0927 03:26:44.763210 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:44.763212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:44.763226 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:44.763276 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:44.763894 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.764397 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.765379 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.765586 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.765642 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:44.767695 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:44.767696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:44.767697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:44.767704 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:44.769284 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:44.769679 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:44.769680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:44.769680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:44.769744 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:44.770004 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:44.770006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:44.770006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:44.770142 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:44.770201 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:44.770503 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 03:26:44.770588 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:26:44.770871 18772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:44.761894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34141 (local address 127.0.98.2:52467)
0927 03:26:44.762025 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:44.762061 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:44.763109 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:44.763348 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:44.763351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:44.763620 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:44.763806 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:44.763811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.764487 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.764488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:44.765297 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:44.765299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.765657 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.765658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:44.765780 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:44.766237 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:26:44.766245 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:44.767603 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:44.769373 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:44.769380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:44.769382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:44.769611 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:44.769798 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:44.769798 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:44.769798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:44.769936 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:44.770214 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:44.770221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:44.770356 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 03:26:44.770458 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 03:26:44.770621 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
I20260927 03:26:44.771320 18770 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34141
I20260927 03:26:44.771426 18770 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:44.771612 18770 heartbeater.cc:511] Master 127.0.98.62:34141 requested a full tablet report, sending...
I20260927 03:26:44.772020 18445 ts_manager.cc:194] Registered new tserver with Master: a24dd6614d334e769a3f9468d36eccc3 (127.0.98.2:42881)
I20260927 03:26:44.772521 18445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52467
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:26:44.793501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34141
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:44.873065 18775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:44.873286 18775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:44.873318 18775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:44.874523 18775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:44.874571 18775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:44.874596 18775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:26:44.876152 18775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:44.876379 18775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:44.876608 18775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:44.881491 18781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:44.881527 18780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:44.881546 18775 server_base.cc:1034] running on GCE node
W20260927 03:26:44.881507 18783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:44.881937 18775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:44.882184 18775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:44.883376 18775 hybrid_clock.cc:648] HybridClock initialized: now 1790479604883364 us; error 30 us; skew 500 ppm
Sep 27 03:26:44 dist-test-slave-gw2v krb5kdc[18400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:44.885973 18775 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:26:44.886410 18775 webserver.cc:460] Webserver started at http://127.0.98.3:34451/ using document root <none> and password file <none>
I20260927 03:26:44.886608 18775 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:44.886657 18775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:44.886780 18775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:44.887892 18775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "1cb82703c98f4e808f99ec4015f2d1e0"
format_stamp: "Formatted at 2026-09-27 03:26:44 on dist-test-slave-gw2v"
server_key: "e39d5150289535e71ce0a64e3344f48a"
server_key_iv: "ba10b88774ca7dda0b077231d467f177"
server_key_version: "encryptionkey@0"
I20260927 03:26:44.888214 18775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1cb82703c98f4e808f99ec4015f2d1e0"
format_stamp: "Formatted at 2026-09-27 03:26:44 on dist-test-slave-gw2v"
server_key: "e39d5150289535e71ce0a64e3344f48a"
server_key_iv: "ba10b88774ca7dda0b077231d467f177"
server_key_version: "encryptionkey@0"
I20260927 03:26:44.889580 18775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:26:44.890273 18790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:44.890514 18775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:26:44.890599 18775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "1cb82703c98f4e808f99ec4015f2d1e0"
format_stamp: "Formatted at 2026-09-27 03:26:44 on dist-test-slave-gw2v"
server_key: "e39d5150289535e71ce0a64e3344f48a"
server_key_iv: "ba10b88774ca7dda0b077231d467f177"
server_key_version: "encryptionkey@0"
I20260927 03:26:44.890661 18775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:44.921095 18775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:44.923550 18775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:44.923694 18775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:44.923908 18775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:44.924225 18775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:44.924258 18775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:44.924288 18775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:44.924314 18775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:44.930193 18775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45069
I20260927 03:26:44.930230 18903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45069 every 8 connection(s)
I20260927 03:26:44.930558 18775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:26:44 dist-test-slave-gw2v krb5kdc[18400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:26:44.938611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 18775
I20260927 03:26:44.938706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:26:44.938903 392 external_mini_cluster.cc:1428] Setting key c9b77b7a02bf1fcd36ca8c64196edea0
I20260927 03:26:44.939919 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:44.931304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50263 (local address 127.0.98.62:34141)
0927 03:26:44.931369 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:44.931371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:44.931578 (+ 207us) server_negotiation.cc:407] Connection header received
0927 03:26:44.932526 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:44.932527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:44.932540 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:44.932603 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:44.933248 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.933784 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.934781 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.935069 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.935128 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:44.937236 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:44.937237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:44.937237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:44.937245 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:44.938597 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:44.939036 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:44.939037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:44.939038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:44.939095 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:44.939311 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:44.939312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:44.939312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:44.939429 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:44.939491 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:44.939754 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 03:26:44.939832 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:26:44.940116 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:44.931285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34141 (local address 127.0.98.3:50263)
0927 03:26:44.931448 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:44.931492 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:44.932401 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:44.932667 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:44.932670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:44.932935 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:44.933155 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:44.933159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.933860 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.933861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:44.934698 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:44.934700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.935145 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.935146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:44.935260 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:44.935746 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:26:44.935754 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:44.937122 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:44.938672 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:44.938680 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:44.938682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:44.938975 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:44.939148 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:44.939148 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:44.939149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:44.939257 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:44.939489 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:44.939497 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:44.939637 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 03:26:44.939724 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 03:26:44.939884 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260927 03:26:44.940518 18904 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34141
I20260927 03:26:44.940605 18904 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:44.940750 18904 heartbeater.cc:511] Master 127.0.98.62:34141 requested a full tablet report, sending...
I20260927 03:26:44.941067 18445 ts_manager.cc:194] Registered new tserver with Master: 1cb82703c98f4e808f99ec4015f2d1e0 (127.0.98.3:45069)
I20260927 03:26:44.941404 18445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50263
I20260927 03:26:44.950657 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:26:44.958317 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:44.952117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34842 (local address 127.0.98.62:34141)
0927 03:26:44.952180 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:44.952183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:44.952272 (+ 89us) server_negotiation.cc:407] Connection header received
0927 03:26:44.952346 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:44.952347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:44.952357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:44.952412 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:44.952987 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.953551 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.954279 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.954471 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.954526 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:44.955874 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:44.955875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:44.955875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:44.955884 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:44.957130 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:44.957551 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:44.957553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:44.957553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:44.957605 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:44.957795 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:44.957796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:44.957796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:44.957937 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:44.958031 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:44.958167 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 03:26:44.958227 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:26:44.959833 18445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:26:44.961434 18445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:26:44.967850 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:44.963864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34780 (local address 127.0.98.2:42881)
0927 03:26:44.964091 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:44.964096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:44.964110 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:26:44.964156 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:44.964158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:44.964175 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:44.964251 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:44.964529 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.965603 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.966923 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.967452 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.967516 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:44.967582 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:44.967722 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 03:26:44.967766 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":91,"threads_started":1}
I20260927 03:26:44.967842 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:44.963734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42881 (local address 127.0.0.1:34780)
0927 03:26:44.964024 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:44.964036 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:44.964088 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:44.964357 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:44.964360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:44.964365 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:44.964445 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:44.964447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.965711 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.965712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:44.966838 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:44.966841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.967527 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.967528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:44.967620 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:44.967635 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:26:44.967702 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:26:44.967756 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":35,"thread_start_us":78,"threads_started":1}
I20260927 03:26:44.968065 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:44.963698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34155 (local address 127.0.0.1:53478)
0927 03:26:44.963864 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:44.963907 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:44.963972 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:44.964358 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:44.964360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:44.964363 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:44.964463 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:44.964464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.965501 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.965502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:44.966846 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:44.966847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.967822 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.967823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:44.967892 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:44.967896 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:26:44.967939 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:26:44.967984 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":9,"thread_start_us":64,"threads_started":1}
I20260927 03:26:44.968266 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:44.963898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45069 (local address 127.0.0.1:52546)
0927 03:26:44.964546 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:44.964555 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:44.964598 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:44.964861 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:44.964862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:44.964864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:44.964902 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:44.964903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.965993 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.965994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:44.966893 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:44.966897 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.967822 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.967823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:44.967888 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:44.967892 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:26:44.967939 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:26:44.967983 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260927 03:26:44.968290 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:44.963839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.0.98.1:34155)
0927 03:26:44.964190 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:44.964194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:44.964202 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:26:44.964241 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:44.964243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:44.964259 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:44.964332 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:44.964543 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.965406 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.966942 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.967664 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.967731 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:44.967787 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:44.968179 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 03:26:44.968220 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":82,"threads_started":1}
I20260927 03:26:44.968407 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:44.964108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52546 (local address 127.0.98.3:45069)
0927 03:26:44.964290 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:44.964293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:44.964648 (+ 355us) server_negotiation.cc:407] Connection header received
0927 03:26:44.964690 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:44.964692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:44.964706 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:44.964769 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:44.964982 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.965910 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.966971 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:44.967663 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:44.967731 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:44.967787 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:44.968027 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 03:26:44.968067 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":96,"threads_started":1}
I20260927 03:26:44.968899 18570 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee31a4442ba4dcd941654478523ea68 (DEFAULT_TABLE table=test-table [id=18a19a00610842a18d862edac22383ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:44.968901 18704 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee31a4442ba4dcd941654478523ea68 (DEFAULT_TABLE table=test-table [id=18a19a00610842a18d862edac22383ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:44.969033 18838 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee31a4442ba4dcd941654478523ea68 (DEFAULT_TABLE table=test-table [id=18a19a00610842a18d862edac22383ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:44.969230 18570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee31a4442ba4dcd941654478523ea68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:44.969240 18704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee31a4442ba4dcd941654478523ea68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:44.969303 18838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee31a4442ba4dcd941654478523ea68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:44.971768 18921 tablet_bootstrap.cc:492] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0: Bootstrap starting.
I20260927 03:26:44.972417 18921 tablet_bootstrap.cc:654] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:44.972397 18922 tablet_bootstrap.cc:492] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0: Bootstrap starting.
I20260927 03:26:44.973044 18922 tablet_bootstrap.cc:654] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:44.973055 18921 log.cc:824] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:44.973546 18922 log.cc:824] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:44.973870 18921 tablet_bootstrap.cc:492] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0: No bootstrap required, opened a new log
I20260927 03:26:44.973896 18923 tablet_bootstrap.cc:492] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3: Bootstrap starting.
I20260927 03:26:44.973941 18921 ts_tablet_manager.cc:1397] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:26:44.974493 18923 tablet_bootstrap.cc:654] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:44.975085 18923 log.cc:824] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:44.975657 18922 tablet_bootstrap.cc:492] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0: No bootstrap required, opened a new log
I20260927 03:26:44.975731 18922 ts_tablet_manager.cc:1397] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:26:44.976046 18921 raft_consensus.cc:348] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "a24dd6614d334e769a3f9468d36eccc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42881 } } peers { permanent_uuid: "1cb82703c98f4e808f99ec4015f2d1e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45069 } }
I20260927 03:26:44.976234 18921 raft_consensus.cc:374] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:44.976274 18921 raft_consensus.cc:729] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02cfcc0b0ad94a6596ed63864cc6d7c0, State: Initialized, Role: FOLLOWER
I20260927 03:26:44.976404 18921 consensus_queue.cc:260] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "a24dd6614d334e769a3f9468d36eccc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42881 } } peers { permanent_uuid: "1cb82703c98f4e808f99ec4015f2d1e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45069 } }
I20260927 03:26:44.976694 18921 ts_tablet_manager.cc:1428] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:26:44.976722 18636 heartbeater.cc:503] Master 127.0.98.62:34141 was elected leader, sending a full tablet report...
I20260927 03:26:44.977489 18922 raft_consensus.cc:348] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "a24dd6614d334e769a3f9468d36eccc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42881 } } peers { permanent_uuid: "1cb82703c98f4e808f99ec4015f2d1e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45069 } }
I20260927 03:26:44.977627 18922 raft_consensus.cc:374] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:44.977653 18922 raft_consensus.cc:729] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb82703c98f4e808f99ec4015f2d1e0, State: Initialized, Role: FOLLOWER
I20260927 03:26:44.977759 18922 consensus_queue.cc:260] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "a24dd6614d334e769a3f9468d36eccc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42881 } } peers { permanent_uuid: "1cb82703c98f4e808f99ec4015f2d1e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45069 } }
I20260927 03:26:44.978024 18922 ts_tablet_manager.cc:1428] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:26:44.978081 18904 heartbeater.cc:503] Master 127.0.98.62:34141 was elected leader, sending a full tablet report...
I20260927 03:26:44.978382 18923 tablet_bootstrap.cc:492] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3: No bootstrap required, opened a new log
I20260927 03:26:44.978457 18923 ts_tablet_manager.cc:1397] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 03:26:44.979997 18923 raft_consensus.cc:348] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "a24dd6614d334e769a3f9468d36eccc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42881 } } peers { permanent_uuid: "1cb82703c98f4e808f99ec4015f2d1e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45069 } }
I20260927 03:26:44.980126 18923 raft_consensus.cc:374] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:44.980146 18923 raft_consensus.cc:729] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a24dd6614d334e769a3f9468d36eccc3, State: Initialized, Role: FOLLOWER
I20260927 03:26:44.980224 18923 consensus_queue.cc:260] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "a24dd6614d334e769a3f9468d36eccc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42881 } } peers { permanent_uuid: "1cb82703c98f4e808f99ec4015f2d1e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45069 } }
I20260927 03:26:44.980459 18923 ts_tablet_manager.cc:1428] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:26:44.980505 18770 heartbeater.cc:503] Master 127.0.98.62:34141 was elected leader, sending a full tablet report...
W20260927 03:26:45.011816 18771 tablet.cc:2369] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:26:45.092136 18637 tablet.cc:2369] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:26:45.181161 18905 tablet.cc:2369] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:26:45.198246 18927 raft_consensus.cc:482] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:26:45.198370 18927 raft_consensus.cc:504] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "a24dd6614d334e769a3f9468d36eccc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42881 } } peers { permanent_uuid: "1cb82703c98f4e808f99ec4015f2d1e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45069 } }
I20260927 03:26:45.198675 18927 leader_election.cc:288] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a24dd6614d334e769a3f9468d36eccc3 (127.0.98.2:42881), 1cb82703c98f4e808f99ec4015f2d1e0 (127.0.98.3:45069)
I20260927 03:26:45.201800 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.199150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36635 (local address 127.0.98.3:45069)
0927 03:26:45.199205 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:45.199208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:45.199216 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:26:45.199271 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:45.199272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:45.199280 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:45.199319 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:45.199539 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.200011 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.201107 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.201480 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.201538 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.201550 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:45.201689 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 03:26:45.201731 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:26:45.201825 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.198921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45645 (local address 127.0.98.2:42881)
0927 03:26:45.198991 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:45.198993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:45.199104 (+ 111us) server_negotiation.cc:407] Connection header received
0927 03:26:45.199199 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:45.199200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:45.199212 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:45.199241 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:45.199496 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.199978 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.201111 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.201497 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.201567 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.201576 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:45.201707 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 03:26:45.201748 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 03:26:45.201828 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.199066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45069 (local address 127.0.98.1:36635)
0927 03:26:45.199152 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:45.199162 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:45.199216 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:45.199406 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:45.199407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:45.199409 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:45.199471 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.199472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.200092 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.200093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.201016 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.201018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.201556 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.201557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.201625 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.201628 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:45.201673 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:26:45.201720 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":1,"thread_start_us":47,"threads_started":1}
I20260927 03:26:45.201831 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.198874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42881 (local address 127.0.98.1:45645)
0927 03:26:45.199014 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:45.199025 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:45.199108 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:45.199360 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:45.199361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:45.199363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:45.199423 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.199425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.200059 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.200060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.201016 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.201019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.201577 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.201578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.201642 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.201646 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:26:45.201692 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:26:45.201734 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":38,"thread_start_us":82,"threads_started":1}
I20260927 03:26:45.202225 18724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee31a4442ba4dcd941654478523ea68" candidate_uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a24dd6614d334e769a3f9468d36eccc3" is_pre_election: true
I20260927 03:26:45.202248 18858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee31a4442ba4dcd941654478523ea68" candidate_uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb82703c98f4e808f99ec4015f2d1e0" is_pre_election: true
I20260927 03:26:45.202368 18724 raft_consensus.cc:2454] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02cfcc0b0ad94a6596ed63864cc6d7c0 in term 0.
I20260927 03:26:45.202382 18858 raft_consensus.cc:2454] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02cfcc0b0ad94a6596ed63864cc6d7c0 in term 0.
I20260927 03:26:45.202580 18524 leader_election.cc:302] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02cfcc0b0ad94a6596ed63864cc6d7c0, a24dd6614d334e769a3f9468d36eccc3; no voters:
I20260927 03:26:45.202718 18927 raft_consensus.cc:2781] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:26:45.202802 18927 raft_consensus.cc:482] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:26:45.202843 18927 raft_consensus.cc:3037] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:45.203533 18927 raft_consensus.cc:504] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "a24dd6614d334e769a3f9468d36eccc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42881 } } peers { permanent_uuid: "1cb82703c98f4e808f99ec4015f2d1e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45069 } }
I20260927 03:26:45.203642 18927 leader_election.cc:288] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [CANDIDATE]: Term 1 election: Requested vote from peers a24dd6614d334e769a3f9468d36eccc3 (127.0.98.2:42881), 1cb82703c98f4e808f99ec4015f2d1e0 (127.0.98.3:45069)
I20260927 03:26:45.203804 18724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee31a4442ba4dcd941654478523ea68" candidate_uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a24dd6614d334e769a3f9468d36eccc3"
I20260927 03:26:45.203872 18724 raft_consensus.cc:3037] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:45.203864 18858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee31a4442ba4dcd941654478523ea68" candidate_uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb82703c98f4e808f99ec4015f2d1e0"
I20260927 03:26:45.203940 18858 raft_consensus.cc:3037] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:45.204586 18724 raft_consensus.cc:2454] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02cfcc0b0ad94a6596ed63864cc6d7c0 in term 1.
I20260927 03:26:45.204638 18858 raft_consensus.cc:2454] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02cfcc0b0ad94a6596ed63864cc6d7c0 in term 1.
I20260927 03:26:45.204758 18524 leader_election.cc:302] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02cfcc0b0ad94a6596ed63864cc6d7c0, a24dd6614d334e769a3f9468d36eccc3; no voters:
I20260927 03:26:45.204864 18927 raft_consensus.cc:2781] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:45.205003 18927 raft_consensus.cc:686] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [term 1 LEADER]: Becoming Leader. State: Replica: 02cfcc0b0ad94a6596ed63864cc6d7c0, State: Running, Role: LEADER
I20260927 03:26:45.205085 18927 consensus_queue.cc:237] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } } peers { permanent_uuid: "a24dd6614d334e769a3f9468d36eccc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42881 } } peers { permanent_uuid: "1cb82703c98f4e808f99ec4015f2d1e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45069 } }
I20260927 03:26:45.205987 18445 catalog_manager.cc:5495] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02cfcc0b0ad94a6596ed63864cc6d7c0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cfcc0b0ad94a6596ed63864cc6d7c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a24dd6614d334e769a3f9468d36eccc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cb82703c98f4e808f99ec4015f2d1e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45069 } health_report { overall_health: UNKNOWN } } }
I20260927 03:26:45.255465 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.252948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.0.98.1:34155)
0927 03:26:45.253008 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:45.253009 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:45.253017 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:26:45.253053 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:45.253054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:45.253074 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:45.253159 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:26:45.253343 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.253829 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.254628 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.254860 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.254927 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.255092 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:26:45.255184 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:26:45.255264 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:45.255392 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 03:26:45.255403 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:26:45.257444 18858 raft_consensus.cc:1261] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0 [term 1 FOLLOWER]: Refusing update from remote peer 02cfcc0b0ad94a6596ed63864cc6d7c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:26:45.257444 18724 raft_consensus.cc:1261] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3 [term 1 FOLLOWER]: Refusing update from remote peer 02cfcc0b0ad94a6596ed63864cc6d7c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:26:45.257834 18932 consensus_queue.cc:1035] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24dd6614d334e769a3f9468d36eccc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:26:45.257992 18927 consensus_queue.cc:1035] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb82703c98f4e808f99ec4015f2d1e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:26:45.260673 18937 mvcc.cc:205] Tried to move back new op lower bound from 7333804463131250688 to 7333804462920671232. Current Snapshot: MvccSnapshot[applied={T|T < 7333804463131250688}]
I20260927 03:26:45.261634 18936 mvcc.cc:205] Tried to move back new op lower bound from 7333804463131250688 to 7333804462920671232. Current Snapshot: MvccSnapshot[applied={T|T < 7333804463131250688 or (T in {7333804463131250688})}]
I20260927 03:26:45.263324 18445 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:26:45.263469 18445 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:26:45.264256 18445 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 516a0c8b4f714beb9b54a6d85ca790dc: Sending DeleteTablet for 3 replicas of tablet 9ee31a4442ba4dcd941654478523ea68
I20260927 03:26:45.264591 18570 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee31a4442ba4dcd941654478523ea68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53478
I20260927 03:26:45.264671 18704 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee31a4442ba4dcd941654478523ea68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34780
I20260927 03:26:45.264859 18838 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee31a4442ba4dcd941654478523ea68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52546
I20260927 03:26:45.264946 18944 tablet_replica.cc:330] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0: stopping tablet replica
I20260927 03:26:45.265045 18944 raft_consensus.cc:2229] T 9ee31a4442ba4dcd941654478523ea68 P 02cfcc0b0ad94a6596ed63864cc6d7c0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:26:45.265120 18946 tablet_replica.cc:330] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0: stopping tablet replica
I20260927 03:26:45.265120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 18507
I20260927 03:26:45.265272 18946 raft_consensus.cc:2229] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:26:45.265405 18946 raft_consensus.cc:2258] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:45.266243 18946 ts_tablet_manager.cc:1905] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:26:45.268025 18946 ts_tablet_manager.cc:1918] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:26:45.268098 18946 log.cc:1196] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/9ee31a4442ba4dcd941654478523ea68
I20260927 03:26:45.268345 18946 ts_tablet_manager.cc:1939] T 9ee31a4442ba4dcd941654478523ea68 P 1cb82703c98f4e808f99ec4015f2d1e0: Deleting consensus metadata
I20260927 03:26:45.268783 18430 catalog_manager.cc:4841] TS 1cb82703c98f4e808f99ec4015f2d1e0 (127.0.98.3:45069): tablet 9ee31a4442ba4dcd941654478523ea68 (table test-table [id=18a19a00610842a18d862edac22383ef]) successfully deleted
W20260927 03:26:45.270455 18429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34155 (error 108)
I20260927 03:26:45.270529 18945 tablet_replica.cc:330] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3: stopping tablet replica
I20260927 03:26:45.270653 18945 raft_consensus.cc:2229] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:26:45.270767 18945 raft_consensus.cc:2258] T 9ee31a4442ba4dcd941654478523ea68 P a24dd6614d334e769a3f9468d36eccc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:45.270978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 18641
I20260927 03:26:45.271114 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.270977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34155 (local address 127.0.0.1:53490)
0927 03:26:45.271038 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:45.271082 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 03:26:45.272279 18429 catalog_manager.cc:4584] TS 02cfcc0b0ad94a6596ed63864cc6d7c0 (127.0.98.1:34155): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ee31a4442ba4dcd941654478523ea68: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34155: connect: Connection refused (error 111)
I20260927 03:26:45.275214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 18775
I20260927 03:26:45.275316 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.275195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42881 (local address 127.0.0.1:34790)
0927 03:26:45.275250 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:45.275285 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 03:26:45.279860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 18413
2026-09-27T03:26:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2052 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T03:26:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:26:45Z Disabled control of system clock
I20260927 03:26:45.306150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39029
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:26:45.385640 18951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:45.385866 18951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:45.385887 18951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:45.387007 18951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:45.387042 18951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:45.387053 18951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:45.387064 18951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:45.387074 18951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:45.388357 18951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:45.388557 18951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:45.388788 18951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:45.393534 18957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:45.393528 18959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:45.393510 18956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:45.393559 18951 server_base.cc:1034] running on GCE node
I20260927 03:26:45.393970 18951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:45.394250 18951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:45.395386 18951 hybrid_clock.cc:648] HybridClock initialized: now 1790479605395367 us; error 35 us; skew 500 ppm
I20260927 03:26:45.396612 18951 webserver.cc:460] Webserver started at http://127.0.98.62:32835/ using document root <none> and password file <none>
I20260927 03:26:45.396835 18951 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:45.396904 18951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:45.397029 18951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:45.398077 18951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "8c2057be249d484ca418dfb6fdd02283"
format_stamp: "Formatted at 2026-09-27 03:26:45 on dist-test-slave-gw2v"
server_key: "ff55fcba0dcf56c289469249d5634cc5"
server_key_iv: "bd23b06e864f1d0fb61597fde60e1f47"
server_key_version: "encryptionkey@0"
I20260927 03:26:45.398371 18951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c2057be249d484ca418dfb6fdd02283"
format_stamp: "Formatted at 2026-09-27 03:26:45 on dist-test-slave-gw2v"
server_key: "ff55fcba0dcf56c289469249d5634cc5"
server_key_iv: "bd23b06e864f1d0fb61597fde60e1f47"
server_key_version: "encryptionkey@0"
I20260927 03:26:45.399590 18951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:26:45.400278 18965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:45.400415 18951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:26:45.400488 18951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "8c2057be249d484ca418dfb6fdd02283"
format_stamp: "Formatted at 2026-09-27 03:26:45 on dist-test-slave-gw2v"
server_key: "ff55fcba0dcf56c289469249d5634cc5"
server_key_iv: "bd23b06e864f1d0fb61597fde60e1f47"
server_key_version: "encryptionkey@0"
I20260927 03:26:45.400552 18951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:45.409415 18951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:45.409710 18951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:45.409874 18951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:45.413973 18951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39029
I20260927 03:26:45.414009 19017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39029 every 8 connection(s)
I20260927 03:26:45.414346 18951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:45.415414 19018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:45.417693 19018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283: Bootstrap starting.
I20260927 03:26:45.418262 19018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:45.418535 19018 log.cc:824] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:45.419305 19018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283: No bootstrap required, opened a new log
I20260927 03:26:45.420267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 18951
I20260927 03:26:45.420387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:26:45.420578 392 external_mini_cluster.cc:1428] Setting key d57fd69027e57ce8a36cb863ff4966ef
I20260927 03:26:45.421314 19018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2057be249d484ca418dfb6fdd02283" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39029 } }
I20260927 03:26:45.421504 19018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:45.421537 19018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c2057be249d484ca418dfb6fdd02283, State: Initialized, Role: FOLLOWER
I20260927 03:26:45.421660 19018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2057be249d484ca418dfb6fdd02283" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39029 } }
I20260927 03:26:45.421752 19018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:45.421844 19018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:45.421897 19018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:45.422443 19018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2057be249d484ca418dfb6fdd02283" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39029 } }
I20260927 03:26:45.422542 19018 leader_election.cc:302] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c2057be249d484ca418dfb6fdd02283; no voters:
I20260927 03:26:45.422693 19018 leader_election.cc:288] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:26:45.422731 19023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:45.422899 19023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [term 1 LEADER]: Becoming Leader. State: Replica: 8c2057be249d484ca418dfb6fdd02283, State: Running, Role: LEADER
I20260927 03:26:45.422974 19018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:45.423033 19023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2057be249d484ca418dfb6fdd02283" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39029 } }
I20260927 03:26:45.423360 19023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c2057be249d484ca418dfb6fdd02283. Latest consensus state: current_term: 1 leader_uuid: "8c2057be249d484ca418dfb6fdd02283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2057be249d484ca418dfb6fdd02283" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39029 } } }
I20260927 03:26:45.423437 19023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:45.423583 19024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c2057be249d484ca418dfb6fdd02283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2057be249d484ca418dfb6fdd02283" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39029 } } }
I20260927 03:26:45.423733 19024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283 [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:45.423647 19029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:45.424273 19029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:45.426661 19029 catalog_manager.cc:1347] Generated new cluster ID: c9d028095ba749559a7d94482e47e009
I20260927 03:26:45.426702 19029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:26:45.427119 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.421174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50170 (local address 127.0.98.62:39029)
0927 03:26:45.421379 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:45.421388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:45.421417 (+ 29us) server_negotiation.cc:407] Connection header received
0927 03:26:45.421538 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:45.421545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:45.421617 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:45.421772 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:45.421995 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.423742 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.424875 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.425482 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.425594 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.426376 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:45.426379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:45.426380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:45.426582 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:45.426627 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:45.426642 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:45.426712 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:45.426797 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:26:45.426911 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260927 03:26:45.439599 19029 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:26:45.440260 19029 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:45.444375 19029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283: Generated new TSK 0
I20260927 03:26:45.444572 19029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:26:45.447405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39029
--builtin_ntp_servers=127.0.98.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:26:45.529304 19042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:45.529524 19042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:45.529546 19042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:45.530753 19042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:45.530794 19042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:45.530844 19042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:26:45.532294 19042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:45.532554 19042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:45.532857 19042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:45.537931 19047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:45.537936 19050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:45.537935 19048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:45.538414 19042 server_base.cc:1034] running on GCE node
I20260927 03:26:45.538637 19042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:45.538949 19042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:45.540122 19042 hybrid_clock.cc:648] HybridClock initialized: now 1790479605540104 us; error 32 us; skew 500 ppm
I20260927 03:26:45.541637 19042 webserver.cc:460] Webserver started at http://127.0.98.1:44589/ using document root <none> and password file <none>
I20260927 03:26:45.541946 19042 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:45.542012 19042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:45.542145 19042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:45.543583 19042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "10117e9da2f94cd0b04d14e1cddf4057"
format_stamp: "Formatted at 2026-09-27 03:26:45 on dist-test-slave-gw2v"
server_key: "5ebdd06ffcb85dbbe8bf72bc2aedf15c"
server_key_iv: "4016aa568e6684468a9ade030e09025e"
server_key_version: "encryptionkey@0"
I20260927 03:26:45.544028 19042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "10117e9da2f94cd0b04d14e1cddf4057"
format_stamp: "Formatted at 2026-09-27 03:26:45 on dist-test-slave-gw2v"
server_key: "5ebdd06ffcb85dbbe8bf72bc2aedf15c"
server_key_iv: "4016aa568e6684468a9ade030e09025e"
server_key_version: "encryptionkey@0"
I20260927 03:26:45.545687 19042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:26:45.546556 19056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:45.546716 19042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:45.546811 19042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "10117e9da2f94cd0b04d14e1cddf4057"
format_stamp: "Formatted at 2026-09-27 03:26:45 on dist-test-slave-gw2v"
server_key: "5ebdd06ffcb85dbbe8bf72bc2aedf15c"
server_key_iv: "4016aa568e6684468a9ade030e09025e"
server_key_version: "encryptionkey@0"
I20260927 03:26:45.546902 19042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:45.569043 19042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:45.569337 19042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:45.569446 19042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:45.569643 19042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:45.569943 19042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:45.569972 19042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:45.569994 19042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:45.570004 19042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:45.575445 19042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44395
I20260927 03:26:45.575501 19169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44395 every 8 connection(s)
I20260927 03:26:45.575826 19042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:26:45.581223 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.576620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33783 (local address 127.0.98.62:39029)
0927 03:26:45.576667 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:45.576669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:45.576952 (+ 283us) server_negotiation.cc:407] Connection header received
0927 03:26:45.577505 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:45.577506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:45.577521 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:45.577576 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:45.578303 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.578803 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.579827 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.580020 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.580077 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.580956 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:45.580957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:45.580958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:45.580963 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:45.580977 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:45.580981 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:45.581022 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:45.581100 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:26:45.581136 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:26:45.581444 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.576567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39029 (local address 127.0.98.1:33783)
0927 03:26:45.576822 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:45.576861 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:45.577395 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:45.577672 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:45.577676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:45.577685 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:45.578206 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.578212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.578919 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.578920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.579730 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.579732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.580099 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.580100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.580233 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.580800 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:26:45.580801 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:45.580834 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:26:45.580860 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:45.581028 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:45.581034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:45.581037 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:45.581100 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:26:45.581212 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
I20260927 03:26:45.581944 19170 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39029
I20260927 03:26:45.582044 19170 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:45.582239 19170 heartbeater.cc:511] Master 127.0.98.62:39029 requested a full tablet report, sending...
I20260927 03:26:45.582429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 19042
I20260927 03:26:45.582552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:26:45.582770 392 external_mini_cluster.cc:1428] Setting key 7497fa45d6927791c295589600c7db76
I20260927 03:26:45.582937 18982 ts_manager.cc:194] Registered new tserver with Master: 10117e9da2f94cd0b04d14e1cddf4057 (127.0.98.1:44395)
I20260927 03:26:45.583859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39029
--builtin_ntp_servers=127.0.98.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:26:45.583860 18982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:33783
W20260927 03:26:45.663800 19173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:45.664079 19173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:45.664111 19173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:45.665668 19173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:45.665715 19173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:45.665750 19173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:26:45.667306 19173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:45.667562 19173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:45.667853 19173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:45.672667 19181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:45.672667 19179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:45.672794 19173 server_base.cc:1034] running on GCE node
W20260927 03:26:45.672734 19178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:45.673089 19173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:45.673326 19173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:45.674469 19173 hybrid_clock.cc:648] HybridClock initialized: now 1790479605674449 us; error 35 us; skew 500 ppm
I20260927 03:26:45.675661 19173 webserver.cc:460] Webserver started at http://127.0.98.2:36293/ using document root <none> and password file <none>
I20260927 03:26:45.675859 19173 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:45.675899 19173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:45.676019 19173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:45.677028 19173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "aa3015bf0a7048a49d3c199fe79807a7"
format_stamp: "Formatted at 2026-09-27 03:26:45 on dist-test-slave-gw2v"
server_key: "4af23f221cf45029f9e8c58ad026a9d3"
server_key_iv: "d03cd874fcbb343294f6506408c3656c"
server_key_version: "encryptionkey@0"
I20260927 03:26:45.677310 19173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "aa3015bf0a7048a49d3c199fe79807a7"
format_stamp: "Formatted at 2026-09-27 03:26:45 on dist-test-slave-gw2v"
server_key: "4af23f221cf45029f9e8c58ad026a9d3"
server_key_iv: "d03cd874fcbb343294f6506408c3656c"
server_key_version: "encryptionkey@0"
I20260927 03:26:45.678442 19173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:26:45.679083 19187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:45.679214 19173 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:26:45.679291 19173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "aa3015bf0a7048a49d3c199fe79807a7"
format_stamp: "Formatted at 2026-09-27 03:26:45 on dist-test-slave-gw2v"
server_key: "4af23f221cf45029f9e8c58ad026a9d3"
server_key_iv: "d03cd874fcbb343294f6506408c3656c"
server_key_version: "encryptionkey@0"
I20260927 03:26:45.679352 19173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:45.697063 19173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:45.697372 19173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:45.697504 19173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:45.697729 19173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:45.698060 19173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:45.698094 19173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:45.698124 19173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:45.698141 19173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:45.703644 19173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32905
I20260927 03:26:45.703711 19300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32905 every 8 connection(s)
I20260927 03:26:45.704021 19173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:26:45.708441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 19173
I20260927 03:26:45.708531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:26:45.708715 392 external_mini_cluster.cc:1428] Setting key 60d8150836de7a03d3c2efa0fa0c83f9
I20260927 03:26:45.709290 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.704711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54791 (local address 127.0.98.62:39029)
0927 03:26:45.704801 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:45.704803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:45.704999 (+ 196us) server_negotiation.cc:407] Connection header received
0927 03:26:45.705600 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:45.705601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:45.705612 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:45.705668 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:45.706337 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.706852 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.707898 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.708092 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.708147 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.709051 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:45.709052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:45.709053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:45.709057 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:45.709073 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:45.709077 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:45.709125 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:45.709188 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 03:26:45.709214 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:26:45.709506 19303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.704711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39029 (local address 127.0.98.2:54791)
0927 03:26:45.704885 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:45.704920 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:45.705485 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:45.705722 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:45.705726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:45.705735 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:45.706214 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.706217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.706926 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.706927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.707808 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.707811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.708172 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.708173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.708372 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.708934 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:26:45.708935 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:45.708965 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:26:45.708990 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:45.709135 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:45.709138 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:45.709140 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:26:45.709186 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:26:45.709277 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260927 03:26:45.709705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39029
--builtin_ntp_servers=127.0.98.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:26:45.709961 19301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39029
I20260927 03:26:45.710062 19301 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:45.710238 19301 heartbeater.cc:511] Master 127.0.98.62:39029 requested a full tablet report, sending...
I20260927 03:26:45.710583 18982 ts_manager.cc:194] Registered new tserver with Master: aa3015bf0a7048a49d3c199fe79807a7 (127.0.98.2:32905)
I20260927 03:26:45.711061 18982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54791
W20260927 03:26:45.790942 19304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:45.791143 19304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:45.791172 19304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:45.792317 19304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:45.792356 19304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:45.792388 19304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:26:45.793697 19304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:45.793941 19304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:45.794224 19304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:45.799242 19310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:45.799227 19309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:45.799230 19312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:45.799532 19304 server_base.cc:1034] running on GCE node
I20260927 03:26:45.799757 19304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:45.799978 19304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:45.801123 19304 hybrid_clock.cc:648] HybridClock initialized: now 1790479605801110 us; error 27 us; skew 500 ppm
I20260927 03:26:45.802479 19304 webserver.cc:460] Webserver started at http://127.0.98.3:44539/ using document root <none> and password file <none>
I20260927 03:26:45.802716 19304 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:45.802762 19304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:45.802893 19304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:45.803993 19304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "2d7e3a1279c148898081078ac436c39a"
format_stamp: "Formatted at 2026-09-27 03:26:45 on dist-test-slave-gw2v"
server_key: "4844ec6d383d57f3daa66048e6c20954"
server_key_iv: "d3e478e4cd1013d6a9629738f0e1c33c"
server_key_version: "encryptionkey@0"
I20260927 03:26:45.804281 19304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2d7e3a1279c148898081078ac436c39a"
format_stamp: "Formatted at 2026-09-27 03:26:45 on dist-test-slave-gw2v"
server_key: "4844ec6d383d57f3daa66048e6c20954"
server_key_iv: "d3e478e4cd1013d6a9629738f0e1c33c"
server_key_version: "encryptionkey@0"
I20260927 03:26:45.805560 19304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:26:45.806284 19318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:45.806470 19304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:45.806553 19304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "2d7e3a1279c148898081078ac436c39a"
format_stamp: "Formatted at 2026-09-27 03:26:45 on dist-test-slave-gw2v"
server_key: "4844ec6d383d57f3daa66048e6c20954"
server_key_iv: "d3e478e4cd1013d6a9629738f0e1c33c"
server_key_version: "encryptionkey@0"
I20260927 03:26:45.806610 19304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:45.817670 19304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:45.817922 19304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:45.818019 19304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:45.818205 19304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:45.818475 19304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:45.818504 19304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:45.818531 19304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:45.818547 19304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:45.824452 19304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37781
I20260927 03:26:45.824499 19431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37781 every 8 connection(s)
I20260927 03:26:45.824851 19304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:26:45.829967 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.825550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35365 (local address 127.0.98.62:39029)
0927 03:26:45.825636 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:45.825638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:45.825868 (+ 230us) server_negotiation.cc:407] Connection header received
0927 03:26:45.826371 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:45.826373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:45.826385 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:45.826430 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:45.827167 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.827664 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.828637 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.828835 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.828899 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.829696 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:45.829697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:45.829698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:45.829703 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:45.829719 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:45.829723 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:45.829768 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:45.829844 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 03:26:45.829885 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 03:26:45.830168 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.825544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39029 (local address 127.0.98.3:35365)
0927 03:26:45.825743 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:45.825784 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:45.826263 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:45.826521 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:45.826525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:45.826536 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:45.827074 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.827079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.827746 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.827747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.828552 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.828554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.828904 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.828905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.829035 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.829554 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:26:45.829554 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:45.829584 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:26:45.829613 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:45.829774 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:45.829779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:45.829783 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:26:45.829844 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:26:45.829949 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260927 03:26:45.830653 19432 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39029
I20260927 03:26:45.830742 19432 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:45.830952 19432 heartbeater.cc:511] Master 127.0.98.62:39029 requested a full tablet report, sending...
I20260927 03:26:45.831316 18982 ts_manager.cc:194] Registered new tserver with Master: 2d7e3a1279c148898081078ac436c39a (127.0.98.3:37781)
I20260927 03:26:45.831676 18982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35365
I20260927 03:26:45.834748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 19304
I20260927 03:26:45.834859 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:26:45.835062 392 external_mini_cluster.cc:1428] Setting key 626ec64712177dd9f08c4a62cce8237e
I20260927 03:26:45.836279 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:26:45.840735 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.837660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.0.98.62:39029)
0927 03:26:45.837718 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:45.837720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:45.837832 (+ 112us) server_negotiation.cc:407] Connection header received
0927 03:26:45.837886 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:45.837887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:45.837899 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:45.837947 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:45.838087 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.838586 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.839489 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.839673 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.839729 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.840483 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:45.840484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:45.840485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:45.840489 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:45.840502 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:45.840506 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:45.840587 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:45.840645 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 03:26:45.840655 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:26:45.842677 18982 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:26:45.844272 18982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:26:45.850585 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.846442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32905 (local address 127.0.0.1:55716)
0927 03:26:45.846624 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:45.846634 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:45.846699 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:45.847193 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:45.847196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:45.847199 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:45.847302 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.847303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.848659 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.848660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.849435 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.849437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.850293 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.850294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.850371 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.850386 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:26:45.850449 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:26:45.850503 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":20,"thread_start_us":98,"threads_started":1}
I20260927 03:26:45.850715 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.846865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.0.98.1:44395)
0927 03:26:45.847046 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:45.847049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:45.847057 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:26:45.847092 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:45.847093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:45.847106 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:45.847179 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:45.847381 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.848536 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.849758 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.850388 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.850456 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.850541 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:45.850604 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 03:26:45.850640 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260927 03:26:45.850724 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.846645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44395 (local address 127.0.0.1:51394)
0927 03:26:45.846870 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:45.846879 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:45.846918 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:45.847198 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:45.847199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:45.847200 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:45.847292 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.847294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.848626 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.848627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.849676 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.849678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.850476 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.850477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.850538 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.850543 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:26:45.850593 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:26:45.850633 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":3,"thread_start_us":65,"threads_started":1}
I20260927 03:26:45.850952 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.846585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.0.98.2:32905)
0927 03:26:45.847017 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:45.847022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:45.847032 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:26:45.847071 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:45.847073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:45.847089 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:45.847170 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:45.847382 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.848582 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.849511 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.850187 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.850254 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.850344 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:45.850794 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 03:26:45.850872 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":105,"threads_started":1}
I20260927 03:26:45.850733 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.846440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37781 (local address 127.0.0.1:38274)
0927 03:26:45.846594 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:45.846633 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:45.846702 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:45.847441 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:45.847442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:45.847444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:45.847487 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.847489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.848372 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.848373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.849607 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:45.849610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.850565 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.850565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:45.850619 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.850621 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:26:45.850647 (+ 26us) client_negotiation.cc:240] Negotiation successful
0927 03:26:45.850677 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260927 03:26:45.851589 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:45.846860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.0.98.3:37781)
0927 03:26:45.847288 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:45.847292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:45.847301 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:26:45.847335 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:45.847336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:45.847351 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:45.847556 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:45.847616 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.848295 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.849688 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:45.850479 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:45.851353 (+ 874us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:45.851453 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:45.851478 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 03:26:45.851529 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":93,"threads_started":1}
I20260927 03:26:45.851454 19104 tablet_service.cc:1467] Processing CreateTablet for tablet b909f89a9f3d47a9892ce2bd3ea1d09f (DEFAULT_TABLE table=test-table [id=29be9e87084a4d3dbd34266921012910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:45.851528 19235 tablet_service.cc:1467] Processing CreateTablet for tablet b909f89a9f3d47a9892ce2bd3ea1d09f (DEFAULT_TABLE table=test-table [id=29be9e87084a4d3dbd34266921012910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:45.851821 19104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b909f89a9f3d47a9892ce2bd3ea1d09f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:45.851821 19235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b909f89a9f3d47a9892ce2bd3ea1d09f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:45.852224 19366 tablet_service.cc:1467] Processing CreateTablet for tablet b909f89a9f3d47a9892ce2bd3ea1d09f (DEFAULT_TABLE table=test-table [id=29be9e87084a4d3dbd34266921012910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:45.852542 19366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b909f89a9f3d47a9892ce2bd3ea1d09f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:45.855212 19447 tablet_bootstrap.cc:492] T b909f89a9f3d47a9892ce2bd3ea1d09f P 10117e9da2f94cd0b04d14e1cddf4057: Bootstrap starting.
I20260927 03:26:45.855248 19446 tablet_bootstrap.cc:492] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a: Bootstrap starting.
I20260927 03:26:45.855278 19448 tablet_bootstrap.cc:492] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7: Bootstrap starting.
I20260927 03:26:45.855832 19448 tablet_bootstrap.cc:654] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:45.855862 19447 tablet_bootstrap.cc:654] T b909f89a9f3d47a9892ce2bd3ea1d09f P 10117e9da2f94cd0b04d14e1cddf4057: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:45.855862 19446 tablet_bootstrap.cc:654] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:45.856132 19448 log.cc:824] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:45.856166 19447 log.cc:824] T b909f89a9f3d47a9892ce2bd3ea1d09f P 10117e9da2f94cd0b04d14e1cddf4057: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:45.856169 19446 log.cc:824] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:45.856834 19446 tablet_bootstrap.cc:492] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a: No bootstrap required, opened a new log
I20260927 03:26:45.856834 19448 tablet_bootstrap.cc:492] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7: No bootstrap required, opened a new log
I20260927 03:26:45.856904 19446 ts_tablet_manager.cc:1397] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:26:45.856904 19448 ts_tablet_manager.cc:1397] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:26:45.856961 19447 tablet_bootstrap.cc:492] T b909f89a9f3d47a9892ce2bd3ea1d09f P 10117e9da2f94cd0b04d14e1cddf4057: No bootstrap required, opened a new log
I20260927 03:26:45.857023 19447 ts_tablet_manager.cc:1397] T b909f89a9f3d47a9892ce2bd3ea1d09f P 10117e9da2f94cd0b04d14e1cddf4057: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:26:45.858978 19447 raft_consensus.cc:348] T b909f89a9f3d47a9892ce2bd3ea1d09f P 10117e9da2f94cd0b04d14e1cddf4057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e3a1279c148898081078ac436c39a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37781 } } peers { permanent_uuid: "aa3015bf0a7048a49d3c199fe79807a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "10117e9da2f94cd0b04d14e1cddf4057" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260927 03:26:45.859126 19447 raft_consensus.cc:374] T b909f89a9f3d47a9892ce2bd3ea1d09f P 10117e9da2f94cd0b04d14e1cddf4057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:45.859159 19447 raft_consensus.cc:729] T b909f89a9f3d47a9892ce2bd3ea1d09f P 10117e9da2f94cd0b04d14e1cddf4057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10117e9da2f94cd0b04d14e1cddf4057, State: Initialized, Role: FOLLOWER
I20260927 03:26:45.859310 19447 consensus_queue.cc:260] T b909f89a9f3d47a9892ce2bd3ea1d09f P 10117e9da2f94cd0b04d14e1cddf4057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e3a1279c148898081078ac436c39a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37781 } } peers { permanent_uuid: "aa3015bf0a7048a49d3c199fe79807a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "10117e9da2f94cd0b04d14e1cddf4057" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260927 03:26:45.859391 19446 raft_consensus.cc:348] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e3a1279c148898081078ac436c39a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37781 } } peers { permanent_uuid: "aa3015bf0a7048a49d3c199fe79807a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "10117e9da2f94cd0b04d14e1cddf4057" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260927 03:26:45.859388 19448 raft_consensus.cc:348] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e3a1279c148898081078ac436c39a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37781 } } peers { permanent_uuid: "aa3015bf0a7048a49d3c199fe79807a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "10117e9da2f94cd0b04d14e1cddf4057" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260927 03:26:45.859558 19446 raft_consensus.cc:374] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:45.859558 19448 raft_consensus.cc:374] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:45.859596 19446 raft_consensus.cc:729] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7e3a1279c148898081078ac436c39a, State: Initialized, Role: FOLLOWER
I20260927 03:26:45.859596 19448 raft_consensus.cc:729] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa3015bf0a7048a49d3c199fe79807a7, State: Initialized, Role: FOLLOWER
I20260927 03:26:45.859594 19447 ts_tablet_manager.cc:1428] T b909f89a9f3d47a9892ce2bd3ea1d09f P 10117e9da2f94cd0b04d14e1cddf4057: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:26:45.859705 19170 heartbeater.cc:503] Master 127.0.98.62:39029 was elected leader, sending a full tablet report...
I20260927 03:26:45.859735 19448 consensus_queue.cc:260] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e3a1279c148898081078ac436c39a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37781 } } peers { permanent_uuid: "aa3015bf0a7048a49d3c199fe79807a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "10117e9da2f94cd0b04d14e1cddf4057" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260927 03:26:45.859736 19446 consensus_queue.cc:260] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e3a1279c148898081078ac436c39a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37781 } } peers { permanent_uuid: "aa3015bf0a7048a49d3c199fe79807a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "10117e9da2f94cd0b04d14e1cddf4057" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260927 03:26:45.859994 19446 ts_tablet_manager.cc:1428] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:26:45.859994 19448 ts_tablet_manager.cc:1428] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:26:45.860114 19301 heartbeater.cc:503] Master 127.0.98.62:39029 was elected leader, sending a full tablet report...
I20260927 03:26:45.860174 19432 heartbeater.cc:503] Master 127.0.98.62:39029 was elected leader, sending a full tablet report...
W20260927 03:26:45.954576 19302 tablet.cc:2369] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:26:46.035548 19453 raft_consensus.cc:482] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:26:46.035686 19453 raft_consensus.cc:504] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e3a1279c148898081078ac436c39a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37781 } } peers { permanent_uuid: "aa3015bf0a7048a49d3c199fe79807a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "10117e9da2f94cd0b04d14e1cddf4057" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260927 03:26:46.036089 19453 leader_election.cc:288] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa3015bf0a7048a49d3c199fe79807a7 (127.0.98.2:32905), 10117e9da2f94cd0b04d14e1cddf4057 (127.0.98.1:44395)
I20260927 03:26:46.039842 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:46.036387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51813 (local address 127.0.98.2:32905)
0927 03:26:46.036440 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:46.036442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:46.036452 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:26:46.036483 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:46.036484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:46.036492 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:46.036556 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:46.036722 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:46.037489 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:46.038941 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:46.039491 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:46.039555 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:46.039565 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:46.039732 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 03:26:46.039768 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:26:46.039862 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:46.036509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57459 (local address 127.0.98.1:44395)
0927 03:26:46.036549 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:46.036550 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:46.036638 (+ 88us) server_negotiation.cc:407] Connection header received
0927 03:26:46.036697 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:46.036697 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:46.036709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:46.036763 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:46.036895 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:46.037557 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:46.038926 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:46.039481 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:46.039555 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:46.039565 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:46.039726 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 03:26:46.039767 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:26:46.039860 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:46.036268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32905 (local address 127.0.98.3:51813)
0927 03:26:46.036333 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:46.036346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:46.036394 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:46.036586 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:46.036587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:46.036588 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:46.036658 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:46.036661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:46.037556 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:46.037557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:46.038845 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:46.038848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:46.039565 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:46.039566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:46.039656 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:46.039660 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:26:46.039715 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:26:46.039769 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":30}
I20260927 03:26:46.040114 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:46.036419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44395 (local address 127.0.98.3:57459)
0927 03:26:46.036585 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:46.036594 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:46.036642 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:46.036776 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:46.036777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:46.036778 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:46.036826 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:46.036827 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:46.037621 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:46.037622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:46.038845 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:46.038848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:46.039566 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:46.039567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:46.039663 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:46.039666 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:46.039715 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:26:46.039767 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":15,"thread_start_us":98,"threads_started":1}
I20260927 03:26:46.040242 19255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909f89a9f3d47a9892ce2bd3ea1d09f" candidate_uuid: "2d7e3a1279c148898081078ac436c39a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa3015bf0a7048a49d3c199fe79807a7" is_pre_election: true
I20260927 03:26:46.040419 19255 raft_consensus.cc:2454] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d7e3a1279c148898081078ac436c39a in term 0.
I20260927 03:26:46.040498 19124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909f89a9f3d47a9892ce2bd3ea1d09f" candidate_uuid: "2d7e3a1279c148898081078ac436c39a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10117e9da2f94cd0b04d14e1cddf4057" is_pre_election: true
I20260927 03:26:46.040627 19322 leader_election.cc:302] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d7e3a1279c148898081078ac436c39a, aa3015bf0a7048a49d3c199fe79807a7; no voters:
I20260927 03:26:46.040656 19124 raft_consensus.cc:2454] T b909f89a9f3d47a9892ce2bd3ea1d09f P 10117e9da2f94cd0b04d14e1cddf4057 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d7e3a1279c148898081078ac436c39a in term 0.
I20260927 03:26:46.040751 19453 raft_consensus.cc:2781] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:26:46.040828 19453 raft_consensus.cc:482] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:26:46.040860 19453 raft_consensus.cc:3037] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:46.041546 19453 raft_consensus.cc:504] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e3a1279c148898081078ac436c39a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37781 } } peers { permanent_uuid: "aa3015bf0a7048a49d3c199fe79807a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "10117e9da2f94cd0b04d14e1cddf4057" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260927 03:26:46.041659 19453 leader_election.cc:288] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [CANDIDATE]: Term 1 election: Requested vote from peers aa3015bf0a7048a49d3c199fe79807a7 (127.0.98.2:32905), 10117e9da2f94cd0b04d14e1cddf4057 (127.0.98.1:44395)
I20260927 03:26:46.041819 19255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909f89a9f3d47a9892ce2bd3ea1d09f" candidate_uuid: "2d7e3a1279c148898081078ac436c39a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa3015bf0a7048a49d3c199fe79807a7"
I20260927 03:26:46.041826 19124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909f89a9f3d47a9892ce2bd3ea1d09f" candidate_uuid: "2d7e3a1279c148898081078ac436c39a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10117e9da2f94cd0b04d14e1cddf4057"
I20260927 03:26:46.041889 19255 raft_consensus.cc:3037] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:46.041891 19124 raft_consensus.cc:3037] T b909f89a9f3d47a9892ce2bd3ea1d09f P 10117e9da2f94cd0b04d14e1cddf4057 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:46.042572 19124 raft_consensus.cc:2454] T b909f89a9f3d47a9892ce2bd3ea1d09f P 10117e9da2f94cd0b04d14e1cddf4057 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d7e3a1279c148898081078ac436c39a in term 1.
I20260927 03:26:46.042624 19255 raft_consensus.cc:2454] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d7e3a1279c148898081078ac436c39a in term 1.
I20260927 03:26:46.042725 19322 leader_election.cc:302] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10117e9da2f94cd0b04d14e1cddf4057, 2d7e3a1279c148898081078ac436c39a; no voters:
I20260927 03:26:46.042804 19453 raft_consensus.cc:2781] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:46.042979 19453 raft_consensus.cc:686] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [term 1 LEADER]: Becoming Leader. State: Replica: 2d7e3a1279c148898081078ac436c39a, State: Running, Role: LEADER
I20260927 03:26:46.043062 19453 consensus_queue.cc:237] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e3a1279c148898081078ac436c39a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37781 } } peers { permanent_uuid: "aa3015bf0a7048a49d3c199fe79807a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } } peers { permanent_uuid: "10117e9da2f94cd0b04d14e1cddf4057" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } }
I20260927 03:26:46.043861 18981 catalog_manager.cc:5495] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d7e3a1279c148898081078ac436c39a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2d7e3a1279c148898081078ac436c39a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e3a1279c148898081078ac436c39a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa3015bf0a7048a49d3c199fe79807a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10117e9da2f94cd0b04d14e1cddf4057" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } health_report { overall_health: UNKNOWN } } }
W20260927 03:26:46.075389 19433 tablet.cc:2369] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:26:46.076483 19171 tablet.cc:2369] T b909f89a9f3d47a9892ce2bd3ea1d09f P 10117e9da2f94cd0b04d14e1cddf4057: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:26:46.078552 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:46.076150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38288 (local address 127.0.98.3:37781)
0927 03:26:46.076199 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:46.076201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:46.076210 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:26:46.076247 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:46.076248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:46.076271 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:46.076333 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:26:46.076480 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:46.077039 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:46.077786 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:46.077994 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:46.078049 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:46.078207 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:26:46.078304 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:26:46.078389 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:46.078445 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 03:26:46.078453 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:26:46.080428 19124 raft_consensus.cc:1261] T b909f89a9f3d47a9892ce2bd3ea1d09f P 10117e9da2f94cd0b04d14e1cddf4057 [term 1 FOLLOWER]: Refusing update from remote peer 2d7e3a1279c148898081078ac436c39a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:26:46.080425 19255 raft_consensus.cc:1261] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7 [term 1 FOLLOWER]: Refusing update from remote peer 2d7e3a1279c148898081078ac436c39a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:26:46.080807 19456 consensus_queue.cc:1035] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa3015bf0a7048a49d3c199fe79807a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:26:46.080948 19453 consensus_queue.cc:1035] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10117e9da2f94cd0b04d14e1cddf4057" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:26:46.083381 19459 mvcc.cc:205] Tried to move back new op lower bound from 7333804466502688768 to 7333804466353156096. Current Snapshot: MvccSnapshot[applied={T|T < 7333804466502688768}]
I20260927 03:26:46.083715 19461 mvcc.cc:205] Tried to move back new op lower bound from 7333804466502688768 to 7333804466353156096. Current Snapshot: MvccSnapshot[applied={T|T < 7333804466502688768}]
I20260927 03:26:46.084270 19460 mvcc.cc:205] Tried to move back new op lower bound from 7333804466502688768 to 7333804466353156096. Current Snapshot: MvccSnapshot[applied={T|T < 7333804466502688768}]
I20260927 03:26:46.085602 18981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:26:46.085701 18981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:26:46.086449 18981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c2057be249d484ca418dfb6fdd02283: Sending DeleteTablet for 3 replicas of tablet b909f89a9f3d47a9892ce2bd3ea1d09f
I20260927 03:26:46.086810 19366 tablet_service.cc:1514] Processing DeleteTablet for tablet b909f89a9f3d47a9892ce2bd3ea1d09f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:46 UTC) from {username='slave'} at 127.0.0.1:38274
I20260927 03:26:46.086885 19235 tablet_service.cc:1514] Processing DeleteTablet for tablet b909f89a9f3d47a9892ce2bd3ea1d09f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:46 UTC) from {username='slave'} at 127.0.0.1:55716
I20260927 03:26:46.086901 19104 tablet_service.cc:1514] Processing DeleteTablet for tablet b909f89a9f3d47a9892ce2bd3ea1d09f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:46 UTC) from {username='slave'} at 127.0.0.1:51394
I20260927 03:26:46.087193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 19042
I20260927 03:26:46.087308 19469 tablet_replica.cc:330] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a: stopping tablet replica
I20260927 03:26:46.087484 19469 raft_consensus.cc:2229] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:26:46.087674 19469 raft_consensus.cc:2258] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:46.088140 19469 ts_tablet_manager.cc:1905] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:26:46.089887 19469 ts_tablet_manager.cc:1918] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:26:46.089993 19469 log.cc:1196] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/b909f89a9f3d47a9892ce2bd3ea1d09f
I20260927 03:26:46.090267 19469 ts_tablet_manager.cc:1939] T b909f89a9f3d47a9892ce2bd3ea1d09f P 2d7e3a1279c148898081078ac436c39a: Deleting consensus metadata
I20260927 03:26:46.090721 18966 catalog_manager.cc:4841] TS 2d7e3a1279c148898081078ac436c39a (127.0.98.3:37781): tablet b909f89a9f3d47a9892ce2bd3ea1d09f (table test-table [id=29be9e87084a4d3dbd34266921012910]) successfully deleted
I20260927 03:26:46.094969 19471 tablet_replica.cc:330] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7: stopping tablet replica
I20260927 03:26:46.095127 19471 raft_consensus.cc:2229] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:26:46.095252 19471 raft_consensus.cc:2258] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:46.095729 19471 ts_tablet_manager.cc:1905] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:26:46.097283 19471 ts_tablet_manager.cc:1918] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:26:46.097407 19471 log.cc:1196] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/b909f89a9f3d47a9892ce2bd3ea1d09f
I20260927 03:26:46.097656 19471 ts_tablet_manager.cc:1939] T b909f89a9f3d47a9892ce2bd3ea1d09f P aa3015bf0a7048a49d3c199fe79807a7: Deleting consensus metadata
I20260927 03:26:46.098114 18969 catalog_manager.cc:4841] TS aa3015bf0a7048a49d3c199fe79807a7 (127.0.98.2:32905): tablet b909f89a9f3d47a9892ce2bd3ea1d09f (table test-table [id=29be9e87084a4d3dbd34266921012910]) successfully deleted
W20260927 03:26:46.098975 18969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44395 (error 108)
I20260927 03:26:46.099116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 19173
I20260927 03:26:46.100037 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:46.099917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44395 (local address 127.0.0.1:51404)
0927 03:26:46.099969 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:46.100008 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:26:46.100153 18969 catalog_manager.cc:4584] TS 10117e9da2f94cd0b04d14e1cddf4057 (127.0.98.1:44395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b909f89a9f3d47a9892ce2bd3ea1d09f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44395: connect: Connection refused (error 111)
I20260927 03:26:46.104892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 19304
I20260927 03:26:46.109061 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 18951
2026-09-27T03:26:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (828 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:26:46 dist-test-slave-gw2v krb5kdc[19475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:26:46 dist-test-slave-gw2v krb5kdc[19475](info): set up 2 sockets
Sep 27 03:26:46 dist-test-slave-gw2v krb5kdc[19475](info): commencing operation
krb5kdc: starting...
W20260927 03:26:47.074509 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:26:47 dist-test-slave-gw2v krb5kdc[19475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:26:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:26:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:26:47.190267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44375
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:47.272378 19488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:47.272596 19488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:47.272619 19488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:47.273758 19488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:47.273800 19488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:47.273813 19488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:47.273825 19488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:47.273836 19488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:47.275331 19488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:47.275563 19488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:47.275784 19488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:47.280504 19496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:47.280561 19494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:47.280740 19488 server_base.cc:1034] running on GCE node
W20260927 03:26:47.280563 19493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:47.281055 19488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:47.281307 19488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:47.282469 19488 hybrid_clock.cc:648] HybridClock initialized: now 1790479607282455 us; error 29 us; skew 500 ppm
Sep 27 03:26:47 dist-test-slave-gw2v krb5kdc[19475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:47.288537 19488 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:26:47.288980 19488 webserver.cc:460] Webserver started at http://127.0.98.62:33541/ using document root <none> and password file <none>
I20260927 03:26:47.289173 19488 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:47.289212 19488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:47.289299 19488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:47.290309 19488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "63ec4cf7926a4d2fb8f8c91d710b5216"
format_stamp: "Formatted at 2026-09-27 03:26:47 on dist-test-slave-gw2v"
server_key: "9cb2de5e66325128b6c729b56775f239"
server_key_iv: "595b35447f5eacfa57a9e7755e76e78a"
server_key_version: "encryptionkey@0"
I20260927 03:26:47.290603 19488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "63ec4cf7926a4d2fb8f8c91d710b5216"
format_stamp: "Formatted at 2026-09-27 03:26:47 on dist-test-slave-gw2v"
server_key: "9cb2de5e66325128b6c729b56775f239"
server_key_iv: "595b35447f5eacfa57a9e7755e76e78a"
server_key_version: "encryptionkey@0"
I20260927 03:26:47.291842 19488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:26:47.292547 19503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:47.292812 19488 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:47.292891 19488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "63ec4cf7926a4d2fb8f8c91d710b5216"
format_stamp: "Formatted at 2026-09-27 03:26:47 on dist-test-slave-gw2v"
server_key: "9cb2de5e66325128b6c729b56775f239"
server_key_iv: "595b35447f5eacfa57a9e7755e76e78a"
server_key_version: "encryptionkey@0"
I20260927 03:26:47.292958 19488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:47.303022 19488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:47.304983 19488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:47.305116 19488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:47.308954 19488 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44375
I20260927 03:26:47.308997 19555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44375 every 8 connection(s)
I20260927 03:26:47.309427 19488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:47.310294 19556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:47.312723 19556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216: Bootstrap starting.
I20260927 03:26:47.313331 19556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:47.313607 19556 log.cc:824] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:47.314368 19556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216: No bootstrap required, opened a new log
I20260927 03:26:47.314987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 19488
I20260927 03:26:47.315104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:26:47.315303 392 external_mini_cluster.cc:1428] Setting key b698f4744c187b029ced039f4d5fd813
I20260927 03:26:47.316480 19556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ec4cf7926a4d2fb8f8c91d710b5216" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44375 } }
I20260927 03:26:47.316625 19556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:47.316648 19556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ec4cf7926a4d2fb8f8c91d710b5216, State: Initialized, Role: FOLLOWER
I20260927 03:26:47.316794 19556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ec4cf7926a4d2fb8f8c91d710b5216" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44375 } }
I20260927 03:26:47.316882 19556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:47.316922 19556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:47.316970 19556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:47.318169 19556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ec4cf7926a4d2fb8f8c91d710b5216" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44375 } }
I20260927 03:26:47.318269 19556 leader_election.cc:302] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63ec4cf7926a4d2fb8f8c91d710b5216; no voters:
I20260927 03:26:47.318421 19556 leader_election.cc:288] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:26:47.318472 19561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:47.318672 19561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [term 1 LEADER]: Becoming Leader. State: Replica: 63ec4cf7926a4d2fb8f8c91d710b5216, State: Running, Role: LEADER
I20260927 03:26:47.318678 19556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:47.318894 19561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ec4cf7926a4d2fb8f8c91d710b5216" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44375 } }
I20260927 03:26:47.319873 19562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63ec4cf7926a4d2fb8f8c91d710b5216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ec4cf7926a4d2fb8f8c91d710b5216" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44375 } } }
I20260927 03:26:47.319896 19563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63ec4cf7926a4d2fb8f8c91d710b5216. Latest consensus state: current_term: 1 leader_uuid: "63ec4cf7926a4d2fb8f8c91d710b5216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ec4cf7926a4d2fb8f8c91d710b5216" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44375 } } }
I20260927 03:26:47.319981 19562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:47.319986 19563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216 [sys.catalog]: This master's current role is: LEADER
W20260927 03:26:47.320559 19576 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:26:47.320611 19576 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:26:47.321168 19577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:47.321401 19577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:47.322778 19577 catalog_manager.cc:1347] Generated new cluster ID: f6d5ae1677db4936868c4271d2163bde
I20260927 03:26:47.322846 19577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:26:47 dist-test-slave-gw2v krb5kdc[19475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:26:47.327725 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.315871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.0.98.62:44375)
0927 03:26:47.316030 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:47.316038 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:47.316066 (+ 28us) server_negotiation.cc:407] Connection header received
0927 03:26:47.316475 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:47.316484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:47.316554 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:47.316713 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:47.317465 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.318464 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.320192 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.320697 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.320769 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.323675 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:47.323680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:47.323680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:47.323693 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:47.325661 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:47.326044 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:47.326048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:47.326048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:47.326536 (+ 488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:47.326718 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:47.326719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:47.326719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:47.327096 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:47.327271 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:47.327352 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 03:26:47.327515 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260927 03:26:47.335207 19577 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:26:47.335736 19577 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:47.344458 19577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216: Generated new TSK 0
I20260927 03:26:47.344684 19577 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:26:47.374168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44375
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:47.491297 19582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:47.491503 19582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:47.491526 19582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:47.492717 19582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:47.492763 19582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:47.492789 19582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:26:47.494313 19582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:47.494554 19582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:47.494761 19582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:47.499634 19590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:47.499653 19588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:47.499634 19587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:47.499954 19582 server_base.cc:1034] running on GCE node
I20260927 03:26:47.500178 19582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:47.500523 19582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:47.501703 19582 hybrid_clock.cc:648] HybridClock initialized: now 1790479607501678 us; error 34 us; skew 500 ppm
Sep 27 03:26:47 dist-test-slave-gw2v krb5kdc[19475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:47.507833 19582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:26:47.508453 19582 webserver.cc:460] Webserver started at http://127.0.98.1:45661/ using document root <none> and password file <none>
I20260927 03:26:47.508668 19582 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:47.508720 19582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:47.508852 19582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:47.509961 19582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "75948531126b4b9394306cd7277208bd"
format_stamp: "Formatted at 2026-09-27 03:26:47 on dist-test-slave-gw2v"
server_key: "7bdcd595aa98c9b3a9a4966abeaca2b5"
server_key_iv: "dc2a9a81315743a5e69825da94baa700"
server_key_version: "encryptionkey@0"
I20260927 03:26:47.510313 19582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "75948531126b4b9394306cd7277208bd"
format_stamp: "Formatted at 2026-09-27 03:26:47 on dist-test-slave-gw2v"
server_key: "7bdcd595aa98c9b3a9a4966abeaca2b5"
server_key_iv: "dc2a9a81315743a5e69825da94baa700"
server_key_version: "encryptionkey@0"
I20260927 03:26:47.511798 19582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:26:47.512651 19597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:47.512825 19582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:26:47.512912 19582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "75948531126b4b9394306cd7277208bd"
format_stamp: "Formatted at 2026-09-27 03:26:47 on dist-test-slave-gw2v"
server_key: "7bdcd595aa98c9b3a9a4966abeaca2b5"
server_key_iv: "dc2a9a81315743a5e69825da94baa700"
server_key_version: "encryptionkey@0"
I20260927 03:26:47.512977 19582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:47.537706 19582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:47.540194 19582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:47.540355 19582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:47.540602 19582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:47.540925 19582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:47.540961 19582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:47.540990 19582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:47.541007 19582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:47.546564 19582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33581
I20260927 03:26:47.546629 19710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33581 every 8 connection(s)
I20260927 03:26:47.546939 19582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:26:47.552639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 19582
I20260927 03:26:47.552753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:26:47.552959 392 external_mini_cluster.cc:1428] Setting key 51f6ffbf80b2e399838ebc409486889f
Sep 27 03:26:47 dist-test-slave-gw2v krb5kdc[19475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:26:47.557054 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.547742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38371 (local address 127.0.98.62:44375)
0927 03:26:47.547820 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:47.547823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:47.547947 (+ 124us) server_negotiation.cc:407] Connection header received
0927 03:26:47.548984 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:47.548985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:47.549000 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:47.549050 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:47.549702 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.550204 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.551248 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.551453 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.551508 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.553828 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:47.553829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:47.553830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:47.553842 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:47.555464 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:47.556009 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:47.556010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:47.556010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:47.556070 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:47.556416 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:47.556417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:47.556417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:47.556534 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:47.556589 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:47.556908 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 03:26:47.556987 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 03:26:47.557304 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.547623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44375 (local address 127.0.98.1:38371)
0927 03:26:47.547806 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:47.547856 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:47.548870 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:47.549130 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:47.549134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:47.549399 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:47.549607 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.549612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.550290 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.550291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.551160 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.551163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.551525 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.551526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.551655 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.552210 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:26:47.552218 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:47.553726 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:47.555545 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:47.555554 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:47.555557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:47.555918 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:47.556204 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:47.556205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:47.556205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:47.556348 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:47.556590 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:47.556596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:47.556773 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 03:26:47.556885 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 03:26:47.557067 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260927 03:26:47.557853 19711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44375
I20260927 03:26:47.557961 19711 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:47.558185 19711 heartbeater.cc:511] Master 127.0.98.62:44375 requested a full tablet report, sending...
I20260927 03:26:47.558745 19520 ts_manager.cc:194] Registered new tserver with Master: 75948531126b4b9394306cd7277208bd (127.0.98.1:33581)
I20260927 03:26:47.559422 19520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38371
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:26:47.577962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44375
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:47.659337 19716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:47.659546 19716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:47.659569 19716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:47.660815 19716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:47.660868 19716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:47.660903 19716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:26:47.663080 19716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:47.663336 19716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:47.663580 19716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:47.669814 19721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:47.669816 19722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:47.669911 19724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:47.670182 19716 server_base.cc:1034] running on GCE node
I20260927 03:26:47.670432 19716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:47.670691 19716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:47.671837 19716 hybrid_clock.cc:648] HybridClock initialized: now 1790479607671819 us; error 28 us; skew 500 ppm
Sep 27 03:26:47 dist-test-slave-gw2v krb5kdc[19475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:47.674201 19716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:26:47.674649 19716 webserver.cc:460] Webserver started at http://127.0.98.2:39767/ using document root <none> and password file <none>
I20260927 03:26:47.674903 19716 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:47.674950 19716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:47.675076 19716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:47.676141 19716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "71a3f4f91ac547c4ae8186e496f30769"
format_stamp: "Formatted at 2026-09-27 03:26:47 on dist-test-slave-gw2v"
server_key: "c3fea72e1dd5963bcf383a343c3ee178"
server_key_iv: "10b5055bca2cc20e82952086f675e099"
server_key_version: "encryptionkey@0"
I20260927 03:26:47.676461 19716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "71a3f4f91ac547c4ae8186e496f30769"
format_stamp: "Formatted at 2026-09-27 03:26:47 on dist-test-slave-gw2v"
server_key: "c3fea72e1dd5963bcf383a343c3ee178"
server_key_iv: "10b5055bca2cc20e82952086f675e099"
server_key_version: "encryptionkey@0"
I20260927 03:26:47.677763 19716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:26:47.678459 19731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:47.678607 19716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:26:47.678695 19716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "71a3f4f91ac547c4ae8186e496f30769"
format_stamp: "Formatted at 2026-09-27 03:26:47 on dist-test-slave-gw2v"
server_key: "c3fea72e1dd5963bcf383a343c3ee178"
server_key_iv: "10b5055bca2cc20e82952086f675e099"
server_key_version: "encryptionkey@0"
I20260927 03:26:47.678752 19716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:47.693010 19716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:47.695091 19716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:47.695199 19716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:47.695382 19716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:47.695629 19716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:47.695659 19716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:47.695679 19716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:47.695690 19716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:47.700906 19716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44825
I20260927 03:26:47.700970 19844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44825 every 8 connection(s)
I20260927 03:26:47.701242 19716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:26:47.702700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 19716
I20260927 03:26:47.702800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:26:47.702989 392 external_mini_cluster.cc:1428] Setting key e9d48d0437ffbc11e512101e1614cb52
Sep 27 03:26:47 dist-test-slave-gw2v krb5kdc[19475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 03:26:47.710682 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.701866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33721 (local address 127.0.98.62:44375)
0927 03:26:47.701927 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:47.701929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:47.702115 (+ 186us) server_negotiation.cc:407] Connection header received
0927 03:26:47.703117 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:47.703119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:47.703132 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:47.703178 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:47.703882 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.704374 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.705349 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.705536 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.705592 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.707891 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:47.707893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:47.707893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:47.707900 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:47.709276 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:47.709722 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:47.709723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:47.709723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:47.709773 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:47.710029 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:47.710030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:47.710030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:47.710140 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:47.710191 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:47.710519 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 03:26:47.710615 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:26:47.710906 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.701852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44375 (local address 127.0.98.2:33721)
0927 03:26:47.702013 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:47.702047 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:47.703018 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:47.703272 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:47.703276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:47.703603 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:47.703796 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.703801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.704454 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.704455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.705269 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.705272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.705610 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.705610 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.705733 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.706206 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:26:47.706214 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:47.707790 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:47.709361 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:47.709368 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:47.709371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:47.709651 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:47.709832 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:47.709832 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:47.709833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:47.709968 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:47.710208 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:47.710215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:47.710378 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 03:26:47.710488 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 03:26:47.710659 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260927 03:26:47.711333 19845 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44375
I20260927 03:26:47.711416 19845 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:47.711566 19845 heartbeater.cc:511] Master 127.0.98.62:44375 requested a full tablet report, sending...
I20260927 03:26:47.711938 19520 ts_manager.cc:194] Registered new tserver with Master: 71a3f4f91ac547c4ae8186e496f30769 (127.0.98.2:44825)
I20260927 03:26:47.712291 19520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:26:47.727649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44375
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:47.808706 19850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:47.808923 19850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:47.808948 19850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:47.810070 19850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:47.810103 19850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:47.810128 19850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:26:47.811563 19850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:47.811743 19850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:47.811961 19850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:47.816996 19855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:47.816996 19858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:47.817015 19856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:47.817090 19850 server_base.cc:1034] running on GCE node
I20260927 03:26:47.817458 19850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:47.817696 19850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:47.818856 19850 hybrid_clock.cc:648] HybridClock initialized: now 1790479607818838 us; error 29 us; skew 500 ppm
Sep 27 03:26:47 dist-test-slave-gw2v krb5kdc[19475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:47.821453 19850 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:26:47.821890 19850 webserver.cc:460] Webserver started at http://127.0.98.3:44497/ using document root <none> and password file <none>
I20260927 03:26:47.822082 19850 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:47.822124 19850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:47.822245 19850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:47.823314 19850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "29af24bddefc4591909384277569a4a0"
format_stamp: "Formatted at 2026-09-27 03:26:47 on dist-test-slave-gw2v"
server_key: "0da639ae0fef14fb5fcb45d0a472bee5"
server_key_iv: "aaa0bc726215e9ee529b19a3161cb99b"
server_key_version: "encryptionkey@0"
I20260927 03:26:47.823634 19850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "29af24bddefc4591909384277569a4a0"
format_stamp: "Formatted at 2026-09-27 03:26:47 on dist-test-slave-gw2v"
server_key: "0da639ae0fef14fb5fcb45d0a472bee5"
server_key_iv: "aaa0bc726215e9ee529b19a3161cb99b"
server_key_version: "encryptionkey@0"
I20260927 03:26:47.824868 19850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:26:47.825640 19865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:47.825793 19850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:26:47.825866 19850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "29af24bddefc4591909384277569a4a0"
format_stamp: "Formatted at 2026-09-27 03:26:47 on dist-test-slave-gw2v"
server_key: "0da639ae0fef14fb5fcb45d0a472bee5"
server_key_iv: "aaa0bc726215e9ee529b19a3161cb99b"
server_key_version: "encryptionkey@0"
I20260927 03:26:47.825932 19850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:47.853197 19850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:47.855413 19850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:47.855556 19850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:47.855796 19850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:47.856156 19850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:47.856195 19850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:47.856240 19850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:47.856264 19850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:47.861557 19850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41767
I20260927 03:26:47.861618 19978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41767 every 8 connection(s)
I20260927 03:26:47.861914 19850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:26:47.862632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 19850
I20260927 03:26:47.862731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:26:47.862950 392 external_mini_cluster.cc:1428] Setting key 278c138425c53ed175e16ffa8e5894cf
Sep 27 03:26:47 dist-test-slave-gw2v krb5kdc[19475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:26:47.871547 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.862645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50533 (local address 127.0.98.62:44375)
0927 03:26:47.862714 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:47.862716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:47.862932 (+ 216us) server_negotiation.cc:407] Connection header received
0927 03:26:47.864057 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:47.864058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:47.864069 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:47.864111 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:47.864714 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.865242 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.866293 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.866483 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.866539 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.868678 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:47.868679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:47.868680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:47.868686 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:47.870145 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:47.870566 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:47.870568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:47.870568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:47.870628 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:47.870927 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:47.870928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:47.870928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:47.871052 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:47.871116 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:47.871414 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 03:26:47.871481 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:26:47.871672 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.862591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44375 (local address 127.0.98.3:50533)
0927 03:26:47.862765 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:47.862840 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:47.863964 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:47.864173 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:47.864177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:47.864430 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:47.864634 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.864638 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.865332 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.865333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.866188 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.866190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.866552 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.866553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.866680 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.867250 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:26:47.867259 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:47.868570 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:47.870258 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:47.870264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:47.870266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:47.870471 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:47.870713 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:26:47.870714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:26:47.870714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:26:47.870862 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:26:47.871115 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:47.871119 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:47.871263 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 03:26:47.871347 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 03:26:47.871475 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260927 03:26:47.872093 19979 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44375
I20260927 03:26:47.872187 19979 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:47.872357 19979 heartbeater.cc:511] Master 127.0.98.62:44375 requested a full tablet report, sending...
I20260927 03:26:47.872773 19520 ts_manager.cc:194] Registered new tserver with Master: 29af24bddefc4591909384277569a4a0 (127.0.98.3:41767)
I20260927 03:26:47.873150 19520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50533
I20260927 03:26:47.874941 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:26:47.882031 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.876272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34160 (local address 127.0.98.62:44375)
0927 03:26:47.876339 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:47.876341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:47.876421 (+ 80us) server_negotiation.cc:407] Connection header received
0927 03:26:47.876466 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:47.876466 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:47.876474 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:47.876516 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:47.877011 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.877568 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.878311 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.878486 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.878531 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.879741 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:47.879742 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:47.879742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:47.879748 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:47.880987 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:47.881360 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:47.881361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:47.881361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:47.881414 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:47.881568 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:47.881568 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:47.881569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:47.881707 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:47.881778 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:47.881913 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 03:26:47.881969 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:26:47.883659 19520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:26:47.885309 19520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:26:47.891909 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.887957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.0.98.1:33581)
0927 03:26:47.888124 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:47.888129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:47.888142 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:26:47.888283 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:47.888285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:47.888301 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:47.888595 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:47.888678 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.889754 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.891035 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.891513 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.891585 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.891653 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:47.891792 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 03:26:47.891835 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20260927 03:26:47.891961 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.887932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44122 (local address 127.0.98.3:41767)
0927 03:26:47.888230 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:47.888234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:47.888295 (+ 61us) server_negotiation.cc:407] Connection header received
0927 03:26:47.888359 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:47.888361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:47.888376 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:47.888443 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:47.888640 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.889809 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.890913 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.891570 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.891623 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.891673 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:47.891844 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 03:26:47.891886 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":84,"threads_started":1}
I20260927 03:26:47.892215 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.887890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40484 (local address 127.0.98.2:44825)
0927 03:26:47.888050 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:47.888056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:47.888189 (+ 133us) server_negotiation.cc:407] Connection header received
0927 03:26:47.888265 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:47.888267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:47.888284 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:47.888351 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:47.888616 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.889364 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.890761 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.891516 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.891606 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.891679 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:47.892082 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 03:26:47.892156 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260927 03:26:47.891947 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.887718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41767 (local address 127.0.0.1:44122)
0927 03:26:47.888242 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:47.888250 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:47.888297 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:47.888529 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:47.888530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:47.888532 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:47.888569 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.888570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.889886 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.889886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.890805 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.890808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.891707 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.891708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.891767 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.891771 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:26:47.891817 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:26:47.891859 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":31,"thread_start_us":87,"threads_started":1}
I20260927 03:26:47.892352 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.887713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44825 (local address 127.0.0.1:40484)
0927 03:26:47.888112 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:47.888122 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:47.888187 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:47.888382 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:47.888385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:47.888388 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:47.888534 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.888537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.889464 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.889465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.890679 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.890682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.891895 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.891896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.891950 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.891954 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:26:47.892003 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:26:47.892038 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":4,"thread_start_us":54,"threads_started":1}
I20260927 03:26:47.892515 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.887713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33581 (local address 127.0.0.1:47030)
0927 03:26:47.888046 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:47.888088 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:47.888142 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:47.888439 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:47.888440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:47.888442 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:47.888521 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.888523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.889885 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.889886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.890959 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.890961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.891606 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.891607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.891693 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.891707 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:26:47.891770 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:26:47.891821 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":30,"thread_start_us":103,"threads_started":1}
I20260927 03:26:47.893054 19779 tablet_service.cc:1467] Processing CreateTablet for tablet b1ed8d702cc24c51881eec42316583f9 (DEFAULT_TABLE table=test-table [id=c1759850418f4a51897d34a79980da9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:47.893069 19913 tablet_service.cc:1467] Processing CreateTablet for tablet b1ed8d702cc24c51881eec42316583f9 (DEFAULT_TABLE table=test-table [id=c1759850418f4a51897d34a79980da9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:47.893332 19913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1ed8d702cc24c51881eec42316583f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:47.893332 19779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1ed8d702cc24c51881eec42316583f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:47.895257 19645 tablet_service.cc:1467] Processing CreateTablet for tablet b1ed8d702cc24c51881eec42316583f9 (DEFAULT_TABLE table=test-table [id=c1759850418f4a51897d34a79980da9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:47.895557 19645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1ed8d702cc24c51881eec42316583f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:47.896047 19996 tablet_bootstrap.cc:492] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0: Bootstrap starting.
I20260927 03:26:47.896799 19996 tablet_bootstrap.cc:654] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:47.896907 19997 tablet_bootstrap.cc:492] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769: Bootstrap starting.
I20260927 03:26:47.897356 19996 log.cc:824] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:47.897428 19997 tablet_bootstrap.cc:654] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:47.897912 19997 log.cc:824] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:47.898164 19996 tablet_bootstrap.cc:492] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0: No bootstrap required, opened a new log
I20260927 03:26:47.898239 19996 ts_tablet_manager.cc:1397] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:26:47.898558 19997 tablet_bootstrap.cc:492] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769: No bootstrap required, opened a new log
I20260927 03:26:47.898681 19997 ts_tablet_manager.cc:1397] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:26:47.898686 20000 tablet_bootstrap.cc:492] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd: Bootstrap starting.
I20260927 03:26:47.899322 20000 tablet_bootstrap.cc:654] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:47.899866 20000 log.cc:824] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:47.900027 19996 raft_consensus.cc:348] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a3f4f91ac547c4ae8186e496f30769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "75948531126b4b9394306cd7277208bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "29af24bddefc4591909384277569a4a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41767 } }
I20260927 03:26:47.900203 19996 raft_consensus.cc:374] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:47.900241 19996 raft_consensus.cc:729] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29af24bddefc4591909384277569a4a0, State: Initialized, Role: FOLLOWER
I20260927 03:26:47.900386 19996 consensus_queue.cc:260] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a3f4f91ac547c4ae8186e496f30769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "75948531126b4b9394306cd7277208bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "29af24bddefc4591909384277569a4a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41767 } }
I20260927 03:26:47.900600 20000 tablet_bootstrap.cc:492] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd: No bootstrap required, opened a new log
I20260927 03:26:47.900669 20000 ts_tablet_manager.cc:1397] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:26:47.900681 19996 ts_tablet_manager.cc:1428] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:26:47.900799 19979 heartbeater.cc:503] Master 127.0.98.62:44375 was elected leader, sending a full tablet report...
I20260927 03:26:47.900696 19997 raft_consensus.cc:348] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a3f4f91ac547c4ae8186e496f30769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "75948531126b4b9394306cd7277208bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "29af24bddefc4591909384277569a4a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41767 } }
I20260927 03:26:47.900866 19997 raft_consensus.cc:374] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:47.900904 19997 raft_consensus.cc:729] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a3f4f91ac547c4ae8186e496f30769, State: Initialized, Role: FOLLOWER
I20260927 03:26:47.901077 19997 consensus_queue.cc:260] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a3f4f91ac547c4ae8186e496f30769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "75948531126b4b9394306cd7277208bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "29af24bddefc4591909384277569a4a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41767 } }
I20260927 03:26:47.901358 19845 heartbeater.cc:503] Master 127.0.98.62:44375 was elected leader, sending a full tablet report...
I20260927 03:26:47.901345 19997 ts_tablet_manager.cc:1428] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 03:26:47.901561 20003 raft_consensus.cc:482] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:26:47.901633 20003 raft_consensus.cc:504] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a3f4f91ac547c4ae8186e496f30769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "75948531126b4b9394306cd7277208bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "29af24bddefc4591909384277569a4a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41767 } }
I20260927 03:26:47.901945 20003 leader_election.cc:288] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75948531126b4b9394306cd7277208bd (127.0.98.1:33581), 29af24bddefc4591909384277569a4a0 (127.0.98.3:41767)
I20260927 03:26:47.903633 20000 raft_consensus.cc:348] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a3f4f91ac547c4ae8186e496f30769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "75948531126b4b9394306cd7277208bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "29af24bddefc4591909384277569a4a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41767 } }
I20260927 03:26:47.903849 20000 raft_consensus.cc:374] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:47.903887 20000 raft_consensus.cc:729] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75948531126b4b9394306cd7277208bd, State: Initialized, Role: FOLLOWER
I20260927 03:26:47.904021 20000 consensus_queue.cc:260] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a3f4f91ac547c4ae8186e496f30769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "75948531126b4b9394306cd7277208bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "29af24bddefc4591909384277569a4a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41767 } }
I20260927 03:26:47.904294 20000 ts_tablet_manager.cc:1428] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 03:26:47.904299 19711 heartbeater.cc:503] Master 127.0.98.62:44375 was elected leader, sending a full tablet report...
I20260927 03:26:47.905401 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.902117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41767 (local address 127.0.98.2:45769)
0927 03:26:47.902246 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:47.902255 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:47.902305 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:47.902507 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:47.902508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:47.902509 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:47.902596 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.902598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.903407 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.903408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.904391 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.904394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.905157 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.905158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.905229 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.905233 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:26:47.905283 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:26:47.905322 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":12,"thread_start_us":87,"threads_started":1}
I20260927 03:26:47.905414 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.902175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45769 (local address 127.0.98.3:41767)
0927 03:26:47.902326 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:47.902328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:47.902334 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:26:47.902355 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:47.902355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:47.902364 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:47.902408 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:47.902700 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.903335 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.904485 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.905048 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.905112 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.905123 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:47.905302 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 03:26:47.905349 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260927 03:26:47.905475 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.902194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54275 (local address 127.0.98.1:33581)
0927 03:26:47.902232 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:47.902233 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:47.902242 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:26:47.902398 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:47.902399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:47.902408 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:47.902452 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:47.902633 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.903392 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.904613 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.905149 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.905213 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.905223 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:47.905382 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 03:26:47.905421 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:26:47.905490 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.902092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33581 (local address 127.0.98.2:54275)
0927 03:26:47.902199 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:47.902211 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:47.902251 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:47.902493 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:47.902494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:47.902495 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:47.902559 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.902561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.903486 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.903487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.904515 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:47.904518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.905224 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.905225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:47.905310 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.905314 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:26:47.905361 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:26:47.905400 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 03:26:47.905762 19665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ed8d702cc24c51881eec42316583f9" candidate_uuid: "71a3f4f91ac547c4ae8186e496f30769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75948531126b4b9394306cd7277208bd" is_pre_election: true
I20260927 03:26:47.905779 19933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ed8d702cc24c51881eec42316583f9" candidate_uuid: "71a3f4f91ac547c4ae8186e496f30769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29af24bddefc4591909384277569a4a0" is_pre_election: true
I20260927 03:26:47.905882 19933 raft_consensus.cc:2454] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a3f4f91ac547c4ae8186e496f30769 in term 0.
I20260927 03:26:47.905882 19665 raft_consensus.cc:2454] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a3f4f91ac547c4ae8186e496f30769 in term 0.
I20260927 03:26:47.906077 19732 leader_election.cc:302] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a3f4f91ac547c4ae8186e496f30769, 75948531126b4b9394306cd7277208bd; no voters:
I20260927 03:26:47.906208 20003 raft_consensus.cc:2781] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:26:47.906277 20003 raft_consensus.cc:482] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:26:47.906297 20003 raft_consensus.cc:3037] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:47.907025 20003 raft_consensus.cc:504] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a3f4f91ac547c4ae8186e496f30769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "75948531126b4b9394306cd7277208bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "29af24bddefc4591909384277569a4a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41767 } }
I20260927 03:26:47.907145 20003 leader_election.cc:288] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [CANDIDATE]: Term 1 election: Requested vote from peers 75948531126b4b9394306cd7277208bd (127.0.98.1:33581), 29af24bddefc4591909384277569a4a0 (127.0.98.3:41767)
I20260927 03:26:47.907311 19933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ed8d702cc24c51881eec42316583f9" candidate_uuid: "71a3f4f91ac547c4ae8186e496f30769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29af24bddefc4591909384277569a4a0"
I20260927 03:26:47.907351 19665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ed8d702cc24c51881eec42316583f9" candidate_uuid: "71a3f4f91ac547c4ae8186e496f30769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75948531126b4b9394306cd7277208bd"
I20260927 03:26:47.907404 19665 raft_consensus.cc:3037] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:47.907406 19933 raft_consensus.cc:3037] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:47.908004 19665 raft_consensus.cc:2454] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a3f4f91ac547c4ae8186e496f30769 in term 1.
I20260927 03:26:47.908164 19732 leader_election.cc:302] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a3f4f91ac547c4ae8186e496f30769, 75948531126b4b9394306cd7277208bd; no voters:
I20260927 03:26:47.908198 19933 raft_consensus.cc:2454] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a3f4f91ac547c4ae8186e496f30769 in term 1.
I20260927 03:26:47.908253 20003 raft_consensus.cc:2781] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:47.908313 20003 raft_consensus.cc:686] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [term 1 LEADER]: Becoming Leader. State: Replica: 71a3f4f91ac547c4ae8186e496f30769, State: Running, Role: LEADER
I20260927 03:26:47.908394 20003 consensus_queue.cc:237] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a3f4f91ac547c4ae8186e496f30769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "75948531126b4b9394306cd7277208bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "29af24bddefc4591909384277569a4a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41767 } }
I20260927 03:26:47.909214 19520 catalog_manager.cc:5495] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71a3f4f91ac547c4ae8186e496f30769 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "71a3f4f91ac547c4ae8186e496f30769" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a3f4f91ac547c4ae8186e496f30769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75948531126b4b9394306cd7277208bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29af24bddefc4591909384277569a4a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41767 } health_report { overall_health: UNKNOWN } } }
I20260927 03:26:47.919217 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.916797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.0.98.2:44825)
0927 03:26:47.916844 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:47.916846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:47.916854 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:26:47.916901 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:47.916902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:47.916923 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:47.916976 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:26:47.917082 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.917541 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.918266 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:47.918515 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:47.918568 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:47.918745 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:26:47.918960 (+ 215us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:26:47.919041 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:47.919110 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 03:26:47.919136 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:26:47.921190 19933 raft_consensus.cc:1261] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0 [term 1 FOLLOWER]: Refusing update from remote peer 71a3f4f91ac547c4ae8186e496f30769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:26:47.921250 19665 raft_consensus.cc:1261] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd [term 1 FOLLOWER]: Refusing update from remote peer 71a3f4f91ac547c4ae8186e496f30769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:26:47.921526 20004 consensus_queue.cc:1035] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29af24bddefc4591909384277569a4a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:26:47.921694 20003 consensus_queue.cc:1035] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75948531126b4b9394306cd7277208bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:26:47.924399 20011 mvcc.cc:205] Tried to move back new op lower bound from 7333804474042593280 to 7333804473993601024. Current Snapshot: MvccSnapshot[applied={T|T < 7333804474042593280}]
I20260927 03:26:47.925127 20010 mvcc.cc:205] Tried to move back new op lower bound from 7333804474042593280 to 7333804473993601024. Current Snapshot: MvccSnapshot[applied={T|T < 7333804474042593280}]
I20260927 03:26:47.926229 19519 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:26:47.926312 19519 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:26:47.926982 19519 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63ec4cf7926a4d2fb8f8c91d710b5216: Sending DeleteTablet for 3 replicas of tablet b1ed8d702cc24c51881eec42316583f9
I20260927 03:26:47.927349 19645 tablet_service.cc:1514] Processing DeleteTablet for tablet b1ed8d702cc24c51881eec42316583f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47030
I20260927 03:26:47.927430 19913 tablet_service.cc:1514] Processing DeleteTablet for tablet b1ed8d702cc24c51881eec42316583f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44122
I20260927 03:26:47.927574 20018 tablet_replica.cc:330] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd: stopping tablet replica
I20260927 03:26:47.927673 20018 raft_consensus.cc:2229] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:26:47.927691 19779 tablet_service.cc:1514] Processing DeleteTablet for tablet b1ed8d702cc24c51881eec42316583f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40484
I20260927 03:26:47.927670 20019 tablet_replica.cc:330] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0: stopping tablet replica
I20260927 03:26:47.927778 20018 raft_consensus.cc:2258] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:47.927886 20019 raft_consensus.cc:2229] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:26:47.928057 20019 raft_consensus.cc:2258] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:47.928068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 19582
I20260927 03:26:47.928328 20018 ts_tablet_manager.cc:1905] T b1ed8d702cc24c51881eec42316583f9 P 75948531126b4b9394306cd7277208bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:26:47.928653 20019 ts_tablet_manager.cc:1905] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:26:47.930471 20019 ts_tablet_manager.cc:1918] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:26:47.930605 20019 log.cc:1196] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/b1ed8d702cc24c51881eec42316583f9
I20260927 03:26:47.930953 20019 ts_tablet_manager.cc:1939] T b1ed8d702cc24c51881eec42316583f9 P 29af24bddefc4591909384277569a4a0: Deleting consensus metadata
I20260927 03:26:47.931448 20020 tablet_replica.cc:330] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769: stopping tablet replica
I20260927 03:26:47.931566 20020 raft_consensus.cc:2229] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:26:47.931667 20020 raft_consensus.cc:2258] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:47.932107 20020 ts_tablet_manager.cc:1905] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:26:47.933355 19506 catalog_manager.cc:4841] TS 29af24bddefc4591909384277569a4a0 (127.0.98.3:41767): tablet b1ed8d702cc24c51881eec42316583f9 (table test-table [id=c1759850418f4a51897d34a79980da9a]) successfully deleted
I20260927 03:26:47.933631 20020 ts_tablet_manager.cc:1918] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:26:47.933702 20020 log.cc:1196] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/b1ed8d702cc24c51881eec42316583f9
I20260927 03:26:47.933946 20020 ts_tablet_manager.cc:1939] T b1ed8d702cc24c51881eec42316583f9 P 71a3f4f91ac547c4ae8186e496f30769: Deleting consensus metadata
I20260927 03:26:47.934343 19507 catalog_manager.cc:4841] TS 71a3f4f91ac547c4ae8186e496f30769 (127.0.98.2:44825): tablet b1ed8d702cc24c51881eec42316583f9 (table test-table [id=c1759850418f4a51897d34a79980da9a]) successfully deleted
W20260927 03:26:47.934962 19504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33581 (error 108)
I20260927 03:26:47.935220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 19716
I20260927 03:26:47.935889 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:47.935369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33581 (local address 127.0.0.1:47034)
0927 03:26:47.935807 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:47.935853 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416}
W20260927 03:26:47.936022 19504 catalog_manager.cc:4584] TS 75948531126b4b9394306cd7277208bd (127.0.98.1:33581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1ed8d702cc24c51881eec42316583f9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33581: connect: Connection refused (error 111)
I20260927 03:26:47.940285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 19850
I20260927 03:26:47.944527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 19488
2026-09-27T03:26:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1836 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T03:26:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:26:47Z Disabled control of system clock
I20260927 03:26:47.970494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42783
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:26:48.050007 20025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:48.050256 20025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:48.050280 20025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:48.051474 20025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:48.051512 20025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:48.051530 20025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:48.051541 20025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:48.051551 20025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:48.052841 20025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42783
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:48.053050 20025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:48.053293 20025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:48.058082 20031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:48.058092 20030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:48.058197 20033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:48.058311 20025 server_base.cc:1034] running on GCE node
I20260927 03:26:48.058534 20025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:48.058804 20025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:48.059993 20025 hybrid_clock.cc:648] HybridClock initialized: now 1790479608059943 us; error 63 us; skew 500 ppm
I20260927 03:26:48.061319 20025 webserver.cc:460] Webserver started at http://127.0.98.62:44435/ using document root <none> and password file <none>
I20260927 03:26:48.061545 20025 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:48.061597 20025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:48.061733 20025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:48.062876 20025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "11a9cbe94b3749fcabd52ee7b8ae9776"
format_stamp: "Formatted at 2026-09-27 03:26:48 on dist-test-slave-gw2v"
server_key: "df78594a78ffa8222fa2e4e0174df140"
server_key_iv: "57e56be55ca4fa96e4519c3f6876e29d"
server_key_version: "encryptionkey@0"
I20260927 03:26:48.063227 20025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "11a9cbe94b3749fcabd52ee7b8ae9776"
format_stamp: "Formatted at 2026-09-27 03:26:48 on dist-test-slave-gw2v"
server_key: "df78594a78ffa8222fa2e4e0174df140"
server_key_iv: "57e56be55ca4fa96e4519c3f6876e29d"
server_key_version: "encryptionkey@0"
I20260927 03:26:48.064483 20025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:26:48.065212 20039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:48.065387 20025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:26:48.065469 20025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "11a9cbe94b3749fcabd52ee7b8ae9776"
format_stamp: "Formatted at 2026-09-27 03:26:48 on dist-test-slave-gw2v"
server_key: "df78594a78ffa8222fa2e4e0174df140"
server_key_iv: "57e56be55ca4fa96e4519c3f6876e29d"
server_key_version: "encryptionkey@0"
I20260927 03:26:48.065536 20025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:48.075707 20025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:48.076012 20025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:48.076134 20025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:48.079972 20025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42783
I20260927 03:26:48.080013 20091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42783 every 8 connection(s)
I20260927 03:26:48.080348 20025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:48.081270 20092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:48.083604 20092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776: Bootstrap starting.
I20260927 03:26:48.084146 20092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:48.084401 20092 log.cc:824] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:48.084826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 20025
I20260927 03:26:48.084939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:26:48.085079 20092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776: No bootstrap required, opened a new log
I20260927 03:26:48.085101 392 external_mini_cluster.cc:1428] Setting key f552736052d582080588ceca3d67db6a
I20260927 03:26:48.087534 20092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a9cbe94b3749fcabd52ee7b8ae9776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42783 } }
I20260927 03:26:48.087688 20092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:48.087723 20092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11a9cbe94b3749fcabd52ee7b8ae9776, State: Initialized, Role: FOLLOWER
I20260927 03:26:48.087849 20092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a9cbe94b3749fcabd52ee7b8ae9776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42783 } }
I20260927 03:26:48.087926 20092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:48.087961 20092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:48.088003 20092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:48.088680 20092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a9cbe94b3749fcabd52ee7b8ae9776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42783 } }
I20260927 03:26:48.088773 20092 leader_election.cc:302] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11a9cbe94b3749fcabd52ee7b8ae9776; no voters:
I20260927 03:26:48.088923 20092 leader_election.cc:288] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:26:48.088955 20097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:48.089071 20097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [term 1 LEADER]: Becoming Leader. State: Replica: 11a9cbe94b3749fcabd52ee7b8ae9776, State: Running, Role: LEADER
I20260927 03:26:48.089170 20092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:48.089159 20097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a9cbe94b3749fcabd52ee7b8ae9776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42783 } }
I20260927 03:26:48.089469 20097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11a9cbe94b3749fcabd52ee7b8ae9776. Latest consensus state: current_term: 1 leader_uuid: "11a9cbe94b3749fcabd52ee7b8ae9776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a9cbe94b3749fcabd52ee7b8ae9776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42783 } } }
I20260927 03:26:48.089533 20097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:48.089736 20102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:48.089776 20099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11a9cbe94b3749fcabd52ee7b8ae9776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a9cbe94b3749fcabd52ee7b8ae9776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42783 } } }
I20260927 03:26:48.090265 20102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:48.091022 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.085583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.0.98.62:42783)
0927 03:26:48.085794 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:48.085802 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:48.085825 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:26:48.086286 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:48.086293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:48.086350 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:48.086462 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:48.086701 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.087617 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.088669 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.089258 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.089408 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.090145 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:48.090149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:48.090149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:48.090343 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:48.090384 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:48.090398 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:48.090454 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:48.090721 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 03:26:48.090801 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260927 03:26:48.089958 20099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776 [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:48.092260 20102 catalog_manager.cc:1347] Generated new cluster ID: c331460f031f4c4a9b73de1c05fa5184
I20260927 03:26:48.092315 20102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:26:48.123019 20102 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:26:48.123657 20102 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:48.128638 20102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776: Generated new TSK 0
I20260927 03:26:48.128837 20102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:26:48.131253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42783
--builtin_ntp_servers=127.0.98.20:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:26:48.212689 20116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:48.212916 20116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:48.212941 20116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:48.214205 20116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:48.214257 20116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:48.214286 20116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:26:48.215829 20116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:48.216068 20116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:48.216396 20116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:48.221227 20121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:48.221256 20122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:48.221256 20124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:48.221421 20116 server_base.cc:1034] running on GCE node
I20260927 03:26:48.221683 20116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:48.221917 20116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:48.223078 20116 hybrid_clock.cc:648] HybridClock initialized: now 1790479608223059 us; error 31 us; skew 500 ppm
I20260927 03:26:48.224596 20116 webserver.cc:460] Webserver started at http://127.0.98.1:44375/ using document root <none> and password file <none>
I20260927 03:26:48.224874 20116 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:48.224942 20116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:48.225086 20116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:48.226590 20116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/data/instance:
uuid: "db0bf3023e85460bb804be71180c7cf9"
format_stamp: "Formatted at 2026-09-27 03:26:48 on dist-test-slave-gw2v"
server_key: "04217752ef106568e32266d349551346"
server_key_iv: "ed715edaf93059dc2b43ee9b4cd42367"
server_key_version: "encryptionkey@0"
I20260927 03:26:48.227042 20116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/wal/instance:
uuid: "db0bf3023e85460bb804be71180c7cf9"
format_stamp: "Formatted at 2026-09-27 03:26:48 on dist-test-slave-gw2v"
server_key: "04217752ef106568e32266d349551346"
server_key_iv: "ed715edaf93059dc2b43ee9b4cd42367"
server_key_version: "encryptionkey@0"
I20260927 03:26:48.228677 20116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:26:48.229357 20130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:48.229486 20116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:26:48.229604 20116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/wal
uuid: "db0bf3023e85460bb804be71180c7cf9"
format_stamp: "Formatted at 2026-09-27 03:26:48 on dist-test-slave-gw2v"
server_key: "04217752ef106568e32266d349551346"
server_key_iv: "ed715edaf93059dc2b43ee9b4cd42367"
server_key_version: "encryptionkey@0"
I20260927 03:26:48.229674 20116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:48.276391 20116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:48.276777 20116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:48.276919 20116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:48.277184 20116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:48.277611 20116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:48.277663 20116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:48.277702 20116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:48.277727 20116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:48.284714 20116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44487
I20260927 03:26:48.284759 20243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44487 every 8 connection(s)
I20260927 03:26:48.285084 20116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:26:48.286218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 20116
I20260927 03:26:48.286306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:26:48.286461 392 external_mini_cluster.cc:1428] Setting key 2e0b5d78c53a4f42c9084cf9637f396c
I20260927 03:26:48.287375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42783
--builtin_ntp_servers=127.0.98.20:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:26:48.290385 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.285759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43155 (local address 127.0.98.62:42783)
0927 03:26:48.285813 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:48.285815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:48.286011 (+ 196us) server_negotiation.cc:407] Connection header received
0927 03:26:48.286620 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:48.286621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:48.286650 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:48.286715 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:48.287418 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.287969 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.288924 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.289116 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.289172 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.290015 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:48.290017 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:48.290017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:48.290022 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:48.290037 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:48.290041 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:48.290187 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:48.290273 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 03:26:48.290303 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:26:48.290592 20246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.285754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42783 (local address 127.0.98.1:43155)
0927 03:26:48.285904 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:48.285938 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:48.286507 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:48.286781 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:48.286786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:48.286796 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:48.287336 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.287342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.288055 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.288056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.288846 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.288849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.289186 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.289187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.289320 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.289869 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:26:48.289869 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:48.289899 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:26:48.289925 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:48.290202 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:48.290207 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:48.290210 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:48.290272 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 03:26:48.290381 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
I20260927 03:26:48.291096 20244 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42783
I20260927 03:26:48.291188 20244 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:48.291364 20244 heartbeater.cc:511] Master 127.0.98.62:42783 requested a full tablet report, sending...
I20260927 03:26:48.291862 20055 ts_manager.cc:194] Registered new tserver with Master: db0bf3023e85460bb804be71180c7cf9 (127.0.98.1:44487)
I20260927 03:26:48.292465 20055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:43155
W20260927 03:26:48.367731 20247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:48.367951 20247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:48.367971 20247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:48.369197 20247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:48.369242 20247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:48.369266 20247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:26:48.370607 20247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:48.370853 20247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:48.371119 20247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:48.376372 20255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:48.376380 20253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:48.376365 20252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:48.376389 20247 server_base.cc:1034] running on GCE node
I20260927 03:26:48.376754 20247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:48.376995 20247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:48.378151 20247 hybrid_clock.cc:648] HybridClock initialized: now 1790479608378133 us; error 35 us; skew 500 ppm
I20260927 03:26:48.379442 20247 webserver.cc:460] Webserver started at http://127.0.98.2:44707/ using document root <none> and password file <none>
I20260927 03:26:48.379690 20247 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:48.379738 20247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:48.379845 20247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:48.380862 20247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/data/instance:
uuid: "23c5a4b6bf2840ed81f4cd1a6750672b"
format_stamp: "Formatted at 2026-09-27 03:26:48 on dist-test-slave-gw2v"
server_key: "d2317d19c963e7e5eec92c6e0b7b20cd"
server_key_iv: "0062cebb3803229fdb2143ea06f13d5f"
server_key_version: "encryptionkey@0"
I20260927 03:26:48.381175 20247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/wal/instance:
uuid: "23c5a4b6bf2840ed81f4cd1a6750672b"
format_stamp: "Formatted at 2026-09-27 03:26:48 on dist-test-slave-gw2v"
server_key: "d2317d19c963e7e5eec92c6e0b7b20cd"
server_key_iv: "0062cebb3803229fdb2143ea06f13d5f"
server_key_version: "encryptionkey@0"
I20260927 03:26:48.382386 20247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:26:48.383271 20261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:48.383446 20247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:26:48.383526 20247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/wal
uuid: "23c5a4b6bf2840ed81f4cd1a6750672b"
format_stamp: "Formatted at 2026-09-27 03:26:48 on dist-test-slave-gw2v"
server_key: "d2317d19c963e7e5eec92c6e0b7b20cd"
server_key_iv: "0062cebb3803229fdb2143ea06f13d5f"
server_key_version: "encryptionkey@0"
I20260927 03:26:48.383591 20247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:48.395537 20247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:48.395866 20247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:48.396005 20247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:48.396216 20247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:48.396519 20247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:48.396551 20247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:48.396579 20247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:48.396602 20247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:48.401924 20247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34215
I20260927 03:26:48.401983 20374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34215 every 8 connection(s)
I20260927 03:26:48.402287 20247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:26:48.402573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 20247
I20260927 03:26:48.402678 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:26:48.402905 392 external_mini_cluster.cc:1428] Setting key f81b5733e349cdcfc4e3064421510ae7
I20260927 03:26:48.403793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42783
--builtin_ntp_servers=127.0.98.20:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:26:48.407536 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.403333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35629 (local address 127.0.98.62:42783)
0927 03:26:48.403378 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:48.403380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:48.403392 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:26:48.403900 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:48.403901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:48.403913 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:48.403970 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:48.404649 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.405236 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.406207 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.406406 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.406461 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.407279 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:48.407280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:48.407281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:48.407285 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:48.407300 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:48.407304 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:48.407349 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:48.407420 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 03:26:48.407447 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:26:48.407737 20377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.403092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42783 (local address 127.0.98.2:35629)
0927 03:26:48.403289 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:48.403331 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:48.403800 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:48.404047 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:48.404051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:48.404062 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:48.404571 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.404576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.405324 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.405325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.406126 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.406129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.406474 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.406474 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.406606 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.407145 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:26:48.407146 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:48.407175 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:26:48.407199 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:48.407355 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:48.407360 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:48.407363 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:48.407425 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 03:26:48.407528 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260927 03:26:48.408205 20375 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42783
I20260927 03:26:48.408288 20375 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:48.408460 20375 heartbeater.cc:511] Master 127.0.98.62:42783 requested a full tablet report, sending...
I20260927 03:26:48.408828 20055 ts_manager.cc:194] Registered new tserver with Master: 23c5a4b6bf2840ed81f4cd1a6750672b (127.0.98.2:34215)
I20260927 03:26:48.409335 20055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35629
W20260927 03:26:48.483348 20378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:48.483552 20378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:48.483572 20378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:48.484660 20378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:48.484694 20378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:48.484717 20378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:26:48.486018 20378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:48.486204 20378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:48.486445 20378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:48.491307 20383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:48.491348 20384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:48.491354 20386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:48.491585 20378 server_base.cc:1034] running on GCE node
I20260927 03:26:48.491798 20378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:48.492022 20378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:48.493213 20378 hybrid_clock.cc:648] HybridClock initialized: now 1790479608493201 us; error 26 us; skew 500 ppm
I20260927 03:26:48.494426 20378 webserver.cc:460] Webserver started at http://127.0.98.3:40187/ using document root <none> and password file <none>
I20260927 03:26:48.494675 20378 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:48.494728 20378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:48.494885 20378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:48.496098 20378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/data/instance:
uuid: "1bd3ddcc8d164f4fb4012a2e13120b96"
format_stamp: "Formatted at 2026-09-27 03:26:48 on dist-test-slave-gw2v"
server_key: "271a82d791c0f59236843154db3da593"
server_key_iv: "869148bb74be346bfff62e6b277ae1b7"
server_key_version: "encryptionkey@0"
I20260927 03:26:48.496439 20378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1bd3ddcc8d164f4fb4012a2e13120b96"
format_stamp: "Formatted at 2026-09-27 03:26:48 on dist-test-slave-gw2v"
server_key: "271a82d791c0f59236843154db3da593"
server_key_iv: "869148bb74be346bfff62e6b277ae1b7"
server_key_version: "encryptionkey@0"
I20260927 03:26:48.497655 20378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:26:48.498371 20392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:48.498530 20378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:26:48.498610 20378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/wal
uuid: "1bd3ddcc8d164f4fb4012a2e13120b96"
format_stamp: "Formatted at 2026-09-27 03:26:48 on dist-test-slave-gw2v"
server_key: "271a82d791c0f59236843154db3da593"
server_key_iv: "869148bb74be346bfff62e6b277ae1b7"
server_key_version: "encryptionkey@0"
I20260927 03:26:48.498677 20378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:48.510185 20378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:48.510479 20378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:48.510608 20378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:48.510875 20378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:26:48.511229 20378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:26:48.511265 20378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:48.511293 20378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:26:48.511317 20378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:48.516716 20378 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39311
I20260927 03:26:48.516767 20505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39311 every 8 connection(s)
I20260927 03:26:48.517081 20378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:26:48.518381 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 20378
I20260927 03:26:48.518492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:26:48.518693 392 external_mini_cluster.cc:1428] Setting key 0d30a8fdbbeadfb81cae1b7ef1178fb9
I20260927 03:26:48.522285 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.517788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50165 (local address 127.0.98.62:42783)
0927 03:26:48.517872 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:48.517875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:48.518116 (+ 241us) server_negotiation.cc:407] Connection header received
0927 03:26:48.518618 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:48.518620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:48.518633 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:48.518673 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:48.519458 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.519941 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.520947 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.521145 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.521201 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.522025 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:48.522027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:48.522027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:48.522032 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:48.522049 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:48.522053 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:48.522094 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:48.522172 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:26:48.522200 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 03:26:48.522500 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.517761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42783 (local address 127.0.98.3:50165)
0927 03:26:48.517937 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:48.517971 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:48.518489 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:48.518771 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:48.518776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:48.518785 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:26:48.519371 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.519377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.520029 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.520030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.520864 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.520867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.521220 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.521222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.521379 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.521889 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:26:48.521889 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:26:48.521920 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:26:48.521946 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:26:48.522100 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:26:48.522105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:26:48.522108 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:48.522172 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 03:26:48.522281 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260927 03:26:48.522974 20506 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42783
I20260927 03:26:48.523056 20506 heartbeater.cc:464] Registering TS with master...
I20260927 03:26:48.523231 20506 heartbeater.cc:511] Master 127.0.98.62:42783 requested a full tablet report, sending...
I20260927 03:26:48.523615 20055 ts_manager.cc:194] Registered new tserver with Master: 1bd3ddcc8d164f4fb4012a2e13120b96 (127.0.98.3:39311)
I20260927 03:26:48.523985 20055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50165
I20260927 03:26:48.530581 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:26:48.534864 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.531909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.0.98.62:42783)
0927 03:26:48.531983 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:48.531985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:48.532057 (+ 72us) server_negotiation.cc:407] Connection header received
0927 03:26:48.532141 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:48.532141 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:48.532152 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:48.532193 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:48.532355 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.532933 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.533712 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.533913 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.533965 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.534603 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:48.534604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:48.534604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:48.534610 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:48.534625 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:48.534629 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:48.534677 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:48.534732 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 03:26:48.534754 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 03:26:48.536302 20055 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:26:48.537895 20055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:26:48.544104 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.539971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39311 (local address 127.0.0.1:34740)
0927 03:26:48.540284 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:48.540298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:48.540334 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:48.540612 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:48.540613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:48.540616 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:48.540688 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.540690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.541696 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.541697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.542922 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.542925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.543845 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.543846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.543908 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.543912 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:26:48.543941 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 03:26:48.544005 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":103,"threads_started":1}
I20260927 03:26:48.544198 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.540261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48484 (local address 127.0.98.2:34215)
0927 03:26:48.540734 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:48.540738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:48.540894 (+ 156us) server_negotiation.cc:407] Connection header received
0927 03:26:48.540933 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:48.540935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:48.540948 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:48.541017 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:48.541115 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.542145 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.543223 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.543888 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.543953 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.544041 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:48.544069 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 03:26:48.544108 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":77,"threads_started":1}
I20260927 03:26:48.544277 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.540251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56592 (local address 127.0.98.1:44487)
0927 03:26:48.540462 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:48.540465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:48.540473 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:26:48.540504 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:48.540505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:48.540522 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:48.540582 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:48.540837 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.541983 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.543200 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.543719 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.543783 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.543882 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:48.544150 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 03:26:48.544194 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":85,"threads_started":1}
I20260927 03:26:48.544497 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.540079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34740 (local address 127.0.98.3:39311)
0927 03:26:48.540445 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:48.540449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:48.540460 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:26:48.540496 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:48.540498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:48.540509 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:48.540596 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:48.540770 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.541628 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.543004 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.543747 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.544249 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.544348 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:48.544375 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 03:26:48.544420 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":76,"threads_started":1}
I20260927 03:26:48.544113 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.539965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44487 (local address 127.0.0.1:56592)
0927 03:26:48.540170 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:48.540216 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:48.540277 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:48.540600 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:48.540603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:48.540604 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:48.540673 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.540675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.542093 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.542094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.543132 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.543134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.543803 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.543804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.543882 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.543896 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:26:48.543970 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:26:48.544013 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":59,"threads_started":1}
I20260927 03:26:48.544756 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.540169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34215 (local address 127.0.0.1:48484)
0927 03:26:48.540792 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:48.540801 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:48.540846 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:48.541016 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:48.541017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:48.541019 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:48.541055 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.541057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.542213 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.542214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.543135 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.543138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.543952 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.543953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.544010 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.544013 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:48.544057 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:26:48.544092 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":68,"threads_started":1}
I20260927 03:26:48.545253 20440 tablet_service.cc:1467] Processing CreateTablet for tablet 42a6f02c6310446ab8d871ba172ca48b (DEFAULT_TABLE table=test-table [id=feb351e832da4287bab235ebe9905e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:48.545460 20178 tablet_service.cc:1467] Processing CreateTablet for tablet 42a6f02c6310446ab8d871ba172ca48b (DEFAULT_TABLE table=test-table [id=feb351e832da4287bab235ebe9905e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:48.545608 20440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a6f02c6310446ab8d871ba172ca48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:48.545776 20178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a6f02c6310446ab8d871ba172ca48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:48.547276 20309 tablet_service.cc:1467] Processing CreateTablet for tablet 42a6f02c6310446ab8d871ba172ca48b (DEFAULT_TABLE table=test-table [id=feb351e832da4287bab235ebe9905e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:26:48.547610 20309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a6f02c6310446ab8d871ba172ca48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:48.548452 20520 tablet_bootstrap.cc:492] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9: Bootstrap starting.
I20260927 03:26:48.549057 20520 tablet_bootstrap.cc:654] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:48.549316 20520 log.cc:824] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:48.549618 20521 tablet_bootstrap.cc:492] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96: Bootstrap starting.
I20260927 03:26:48.550079 20520 tablet_bootstrap.cc:492] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9: No bootstrap required, opened a new log
I20260927 03:26:48.550134 20521 tablet_bootstrap.cc:654] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:48.550149 20520 ts_tablet_manager.cc:1397] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:26:48.550411 20521 log.cc:824] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:48.550542 20523 tablet_bootstrap.cc:492] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b: Bootstrap starting.
I20260927 03:26:48.551163 20523 tablet_bootstrap.cc:654] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:48.551169 20521 tablet_bootstrap.cc:492] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96: No bootstrap required, opened a new log
I20260927 03:26:48.551442 20521 ts_tablet_manager.cc:1397] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:26:48.551469 20523 log.cc:824] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:48.552211 20523 tablet_bootstrap.cc:492] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b: No bootstrap required, opened a new log
I20260927 03:26:48.552285 20523 ts_tablet_manager.cc:1397] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:26:48.552388 20520 raft_consensus.cc:348] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd3ddcc8d164f4fb4012a2e13120b96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39311 } } peers { permanent_uuid: "db0bf3023e85460bb804be71180c7cf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44487 } } peers { permanent_uuid: "23c5a4b6bf2840ed81f4cd1a6750672b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34215 } }
I20260927 03:26:48.552521 20520 raft_consensus.cc:374] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:48.552552 20520 raft_consensus.cc:729] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0bf3023e85460bb804be71180c7cf9, State: Initialized, Role: FOLLOWER
I20260927 03:26:48.552685 20520 consensus_queue.cc:260] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd3ddcc8d164f4fb4012a2e13120b96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39311 } } peers { permanent_uuid: "db0bf3023e85460bb804be71180c7cf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44487 } } peers { permanent_uuid: "23c5a4b6bf2840ed81f4cd1a6750672b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34215 } }
I20260927 03:26:48.552922 20520 ts_tablet_manager.cc:1428] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:26:48.553032 20244 heartbeater.cc:503] Master 127.0.98.62:42783 was elected leader, sending a full tablet report...
I20260927 03:26:48.554136 20521 raft_consensus.cc:348] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd3ddcc8d164f4fb4012a2e13120b96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39311 } } peers { permanent_uuid: "db0bf3023e85460bb804be71180c7cf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44487 } } peers { permanent_uuid: "23c5a4b6bf2840ed81f4cd1a6750672b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34215 } }
I20260927 03:26:48.554327 20521 raft_consensus.cc:374] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:48.554361 20521 raft_consensus.cc:729] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd3ddcc8d164f4fb4012a2e13120b96, State: Initialized, Role: FOLLOWER
I20260927 03:26:48.554471 20521 consensus_queue.cc:260] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd3ddcc8d164f4fb4012a2e13120b96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39311 } } peers { permanent_uuid: "db0bf3023e85460bb804be71180c7cf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44487 } } peers { permanent_uuid: "23c5a4b6bf2840ed81f4cd1a6750672b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34215 } }
I20260927 03:26:48.554733 20521 ts_tablet_manager.cc:1428] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:26:48.554767 20506 heartbeater.cc:503] Master 127.0.98.62:42783 was elected leader, sending a full tablet report...
I20260927 03:26:48.554936 20523 raft_consensus.cc:348] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd3ddcc8d164f4fb4012a2e13120b96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39311 } } peers { permanent_uuid: "db0bf3023e85460bb804be71180c7cf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44487 } } peers { permanent_uuid: "23c5a4b6bf2840ed81f4cd1a6750672b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34215 } }
I20260927 03:26:48.555092 20523 raft_consensus.cc:374] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:48.555135 20523 raft_consensus.cc:729] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c5a4b6bf2840ed81f4cd1a6750672b, State: Initialized, Role: FOLLOWER
I20260927 03:26:48.555445 20523 consensus_queue.cc:260] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd3ddcc8d164f4fb4012a2e13120b96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39311 } } peers { permanent_uuid: "db0bf3023e85460bb804be71180c7cf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44487 } } peers { permanent_uuid: "23c5a4b6bf2840ed81f4cd1a6750672b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34215 } }
I20260927 03:26:48.555748 20523 ts_tablet_manager.cc:1428] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 03:26:48.555888 20375 heartbeater.cc:503] Master 127.0.98.62:42783 was elected leader, sending a full tablet report...
I20260927 03:26:48.581542 20528 raft_consensus.cc:482] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:26:48.581684 20528 raft_consensus.cc:504] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd3ddcc8d164f4fb4012a2e13120b96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39311 } } peers { permanent_uuid: "db0bf3023e85460bb804be71180c7cf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44487 } } peers { permanent_uuid: "23c5a4b6bf2840ed81f4cd1a6750672b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34215 } }
I20260927 03:26:48.581996 20528 leader_election.cc:288] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bd3ddcc8d164f4fb4012a2e13120b96 (127.0.98.3:39311), db0bf3023e85460bb804be71180c7cf9 (127.0.98.1:44487)
I20260927 03:26:48.584647 20377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.582178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39311 (local address 127.0.98.2:41967)
0927 03:26:48.582241 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:48.582251 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:48.582293 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:48.582413 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:48.582414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:48.582415 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:48.582496 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.582497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.583162 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.583163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.583894 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.583896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.584407 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.584408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.584478 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.584481 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:48.584524 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:26:48.584566 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 03:26:48.584643 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.582245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41967 (local address 127.0.98.3:39311)
0927 03:26:48.582308 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:48.582310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:48.582319 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:26:48.582351 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:48.582352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:48.582360 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:48.582404 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:48.582554 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.583062 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.583979 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.584336 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.584395 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.584406 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:48.584536 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 03:26:48.584572 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:26:48.585052 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.582179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44487 (local address 127.0.98.2:51525)
0927 03:26:48.582493 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:48.582504 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:26:48.582576 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:26:48.582727 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:26:48.582728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:26:48.582729 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:26:48.582803 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.582805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.583474 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.583475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.584236 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:26:48.584238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.584795 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.584796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:26:48.584866 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.584869 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:26:48.584945 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 03:26:48.584982 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":112,"thread_start_us":128,"threads_started":1}
I20260927 03:26:48.585071 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.582251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51525 (local address 127.0.98.1:44487)
0927 03:26:48.582325 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:48.582327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:48.582580 (+ 253us) server_negotiation.cc:407] Connection header received
0927 03:26:48.582650 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:48.582651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:48.582663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:48.582720 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:26:48.582900 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.583391 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.584323 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.584699 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.584769 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.584779 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:48.584959 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 03:26:48.585000 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 03:26:48.585071 20460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a6f02c6310446ab8d871ba172ca48b" candidate_uuid: "23c5a4b6bf2840ed81f4cd1a6750672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd3ddcc8d164f4fb4012a2e13120b96" is_pre_election: true
I20260927 03:26:48.585237 20460 raft_consensus.cc:2454] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23c5a4b6bf2840ed81f4cd1a6750672b in term 0.
I20260927 03:26:48.585443 20198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a6f02c6310446ab8d871ba172ca48b" candidate_uuid: "23c5a4b6bf2840ed81f4cd1a6750672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0bf3023e85460bb804be71180c7cf9" is_pre_election: true
I20260927 03:26:48.585502 20265 leader_election.cc:302] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bd3ddcc8d164f4fb4012a2e13120b96, 23c5a4b6bf2840ed81f4cd1a6750672b; no voters:
I20260927 03:26:48.585618 20198 raft_consensus.cc:2454] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23c5a4b6bf2840ed81f4cd1a6750672b in term 0.
I20260927 03:26:48.585635 20528 raft_consensus.cc:2781] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:26:48.585678 20528 raft_consensus.cc:482] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:26:48.585708 20528 raft_consensus.cc:3037] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:48.586437 20528 raft_consensus.cc:504] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd3ddcc8d164f4fb4012a2e13120b96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39311 } } peers { permanent_uuid: "db0bf3023e85460bb804be71180c7cf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44487 } } peers { permanent_uuid: "23c5a4b6bf2840ed81f4cd1a6750672b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34215 } }
I20260927 03:26:48.586560 20528 leader_election.cc:288] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [CANDIDATE]: Term 1 election: Requested vote from peers 1bd3ddcc8d164f4fb4012a2e13120b96 (127.0.98.3:39311), db0bf3023e85460bb804be71180c7cf9 (127.0.98.1:44487)
I20260927 03:26:48.586730 20198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a6f02c6310446ab8d871ba172ca48b" candidate_uuid: "23c5a4b6bf2840ed81f4cd1a6750672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0bf3023e85460bb804be71180c7cf9"
I20260927 03:26:48.586804 20198 raft_consensus.cc:3037] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:48.586849 20460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a6f02c6310446ab8d871ba172ca48b" candidate_uuid: "23c5a4b6bf2840ed81f4cd1a6750672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd3ddcc8d164f4fb4012a2e13120b96"
I20260927 03:26:48.586920 20460 raft_consensus.cc:3037] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:48.587569 20198 raft_consensus.cc:2454] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23c5a4b6bf2840ed81f4cd1a6750672b in term 1.
I20260927 03:26:48.587610 20460 raft_consensus.cc:2454] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23c5a4b6bf2840ed81f4cd1a6750672b in term 1.
I20260927 03:26:48.587745 20263 leader_election.cc:302] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23c5a4b6bf2840ed81f4cd1a6750672b, db0bf3023e85460bb804be71180c7cf9; no voters:
I20260927 03:26:48.587867 20528 raft_consensus.cc:2781] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:48.588024 20528 raft_consensus.cc:686] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [term 1 LEADER]: Becoming Leader. State: Replica: 23c5a4b6bf2840ed81f4cd1a6750672b, State: Running, Role: LEADER
I20260927 03:26:48.588116 20528 consensus_queue.cc:237] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd3ddcc8d164f4fb4012a2e13120b96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39311 } } peers { permanent_uuid: "db0bf3023e85460bb804be71180c7cf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44487 } } peers { permanent_uuid: "23c5a4b6bf2840ed81f4cd1a6750672b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34215 } }
I20260927 03:26:48.588943 20056 catalog_manager.cc:5495] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b reported cstate change: term changed from 0 to 1, leader changed from <none> to 23c5a4b6bf2840ed81f4cd1a6750672b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "23c5a4b6bf2840ed81f4cd1a6750672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd3ddcc8d164f4fb4012a2e13120b96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0bf3023e85460bb804be71180c7cf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23c5a4b6bf2840ed81f4cd1a6750672b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34215 } health_report { overall_health: HEALTHY } } }
I20260927 03:26:48.602068 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.599742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.0.98.2:34215)
0927 03:26:48.599808 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:48.599811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:48.599818 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:26:48.599859 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:48.599860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:48.599881 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:48.599940 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:26:48.600075 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.600551 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.601313 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:48.601525 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:48.601577 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:48.601747 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:26:48.601852 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:26:48.601924 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:48.601943 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 03:26:48.601964 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:26:48.603824 20460 raft_consensus.cc:1261] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96 [term 1 FOLLOWER]: Refusing update from remote peer 23c5a4b6bf2840ed81f4cd1a6750672b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:26:48.603858 20198 raft_consensus.cc:1261] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9 [term 1 FOLLOWER]: Refusing update from remote peer 23c5a4b6bf2840ed81f4cd1a6750672b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:26:48.604185 20528 consensus_queue.cc:1035] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0bf3023e85460bb804be71180c7cf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:26:48.604331 20530 consensus_queue.cc:1035] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bd3ddcc8d164f4fb4012a2e13120b96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:26:48.607234 20533 mvcc.cc:205] Tried to move back new op lower bound from 7333804476838887424 to 7333804476777545728. Current Snapshot: MvccSnapshot[applied={T|T < 7333804476838887424 or (T in {7333804476838887424})}]
I20260927 03:26:48.607412 20534 mvcc.cc:205] Tried to move back new op lower bound from 7333804476838887424 to 7333804476777545728. Current Snapshot: MvccSnapshot[applied={T|T < 7333804476838887424}]
I20260927 03:26:48.608954 20056 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:26:48.609040 20056 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:26:48.609745 20056 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11a9cbe94b3749fcabd52ee7b8ae9776: Sending DeleteTablet for 3 replicas of tablet 42a6f02c6310446ab8d871ba172ca48b
I20260927 03:26:48.610047 20309 tablet_service.cc:1514] Processing DeleteTablet for tablet 42a6f02c6310446ab8d871ba172ca48b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:48 UTC) from {username='slave'} at 127.0.0.1:48484
I20260927 03:26:48.610070 20178 tablet_service.cc:1514] Processing DeleteTablet for tablet 42a6f02c6310446ab8d871ba172ca48b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:48 UTC) from {username='slave'} at 127.0.0.1:56592
I20260927 03:26:48.610126 20440 tablet_service.cc:1514] Processing DeleteTablet for tablet 42a6f02c6310446ab8d871ba172ca48b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:26:48 UTC) from {username='slave'} at 127.0.0.1:34740
I20260927 03:26:48.610297 20543 tablet_replica.cc:330] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9: stopping tablet replica
I20260927 03:26:48.610361 20544 tablet_replica.cc:330] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96: stopping tablet replica
I20260927 03:26:48.610399 20543 raft_consensus.cc:2229] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:26:48.610502 20543 raft_consensus.cc:2258] T 42a6f02c6310446ab8d871ba172ca48b P db0bf3023e85460bb804be71180c7cf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:48.610591 20542 tablet_replica.cc:330] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b: stopping tablet replica
I20260927 03:26:48.610769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 20116
I20260927 03:26:48.611256 20542 raft_consensus.cc:2229] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:26:48.611356 20542 raft_consensus.cc:2258] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:48.611689 20542 ts_tablet_manager.cc:1905] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:26:48.613379 20542 ts_tablet_manager.cc:1918] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:26:48.613440 20542 log.cc:1196] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-1/wal/wals/42a6f02c6310446ab8d871ba172ca48b
I20260927 03:26:48.613680 20542 ts_tablet_manager.cc:1939] T 42a6f02c6310446ab8d871ba172ca48b P 23c5a4b6bf2840ed81f4cd1a6750672b: Deleting consensus metadata
I20260927 03:26:48.614023 20041 catalog_manager.cc:4841] TS 23c5a4b6bf2840ed81f4cd1a6750672b (127.0.98.2:34215): tablet 42a6f02c6310446ab8d871ba172ca48b (table test-table [id=feb351e832da4287bab235ebe9905e2a]) successfully deleted
I20260927 03:26:48.614940 20544 raft_consensus.cc:2229] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:26:48.615141 20544 raft_consensus.cc:2258] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:26:48.615521 20544 ts_tablet_manager.cc:1905] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:26:48.616847 20544 ts_tablet_manager.cc:1918] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:26:48.616907 20544 log.cc:1196] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96: Deleting WAL directory at /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479369225731-392-0/minicluster-data/ts-2/wal/wals/42a6f02c6310446ab8d871ba172ca48b
W20260927 03:26:48.617169 20041 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44487 (error 108)
I20260927 03:26:48.617229 20544 ts_tablet_manager.cc:1939] T 42a6f02c6310446ab8d871ba172ca48b P 1bd3ddcc8d164f4fb4012a2e13120b96: Deleting consensus metadata
I20260927 03:26:48.617304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 20247
I20260927 03:26:48.617784 20043 catalog_manager.cc:4841] TS 1bd3ddcc8d164f4fb4012a2e13120b96 (127.0.98.3:39311): tablet 42a6f02c6310446ab8d871ba172ca48b (table test-table [id=feb351e832da4287bab235ebe9905e2a]) successfully deleted
I20260927 03:26:48.618037 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:48.617878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44487 (local address 127.0.0.1:56602)
0927 03:26:48.617965 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 03:26:48.618005 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 03:26:48.618146 20041 catalog_manager.cc:4584] TS db0bf3023e85460bb804be71180c7cf9 (127.0.98.1:44487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42a6f02c6310446ab8d871ba172ca48b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44487: connect: Connection refused (error 111)
I20260927 03:26:48.621985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 20378
I20260927 03:26:48.626072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 20025
2026-09-27T03:26:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (680 ms)
[----------] 6 tests from EncryptionPolicyTest (8311 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:26:48 dist-test-slave-gw2v krb5kdc[20548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:26:48 dist-test-slave-gw2v krb5kdc[20548](info): set up 2 sockets
Sep 27 03:26:48 dist-test-slave-gw2v krb5kdc[20548](info): commencing operation
krb5kdc: starting...
W20260927 03:26:49.582626 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:26:49 dist-test-slave-gw2v krb5kdc[20548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:26:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:26:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:26:49.702891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:49.784097 20561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:49.784313 20561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:49.784334 20561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:49.785494 20561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:49.785542 20561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:49.785555 20561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:49.785567 20561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:49.785578 20561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:49.787092 20561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39999
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:49.787336 20561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:49.787569 20561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:26:49.792542 20561 server_base.cc:1034] running on GCE node
W20260927 03:26:49.792502 20567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:49.792497 20569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:49.792506 20566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:49.792963 20561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:49.793283 20561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:49.794461 20561 hybrid_clock.cc:648] HybridClock initialized: now 1790479609794443 us; error 32 us; skew 500 ppm
Sep 27 03:26:49 dist-test-slave-gw2v krb5kdc[20548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:49.797329 20561 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:26:49.797801 20561 webserver.cc:460] Webserver started at http://127.0.0.1:36487/ using document root <none> and password file <none>
I20260927 03:26:49.798022 20561 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:49.798069 20561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:49.798189 20561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:49.799415 20561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "e8e8977afd234ad79288e944d65a8c8b"
format_stamp: "Formatted at 2026-09-27 03:26:49 on dist-test-slave-gw2v"
server_key: "6cfb49a18468ad2526059a4571dd6153"
server_key_iv: "76b1948410ff2f75dffbef34aea405ed"
server_key_version: "encryptionkey@0"
I20260927 03:26:49.799763 20561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "e8e8977afd234ad79288e944d65a8c8b"
format_stamp: "Formatted at 2026-09-27 03:26:49 on dist-test-slave-gw2v"
server_key: "6cfb49a18468ad2526059a4571dd6153"
server_key_iv: "76b1948410ff2f75dffbef34aea405ed"
server_key_version: "encryptionkey@0"
I20260927 03:26:49.801260 20561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:26:49.802078 20576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:49.802245 20561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:49.802336 20561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "e8e8977afd234ad79288e944d65a8c8b"
format_stamp: "Formatted at 2026-09-27 03:26:49 on dist-test-slave-gw2v"
server_key: "6cfb49a18468ad2526059a4571dd6153"
server_key_iv: "76b1948410ff2f75dffbef34aea405ed"
server_key_version: "encryptionkey@0"
I20260927 03:26:49.802409 20561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:49.813786 20561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:49.815817 20561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:49.815954 20561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:49.820087 20561 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39999
I20260927 03:26:49.820190 20628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39999 every 8 connection(s)
I20260927 03:26:49.820541 20561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:49.821417 20629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:49.823874 20629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b: Bootstrap starting.
I20260927 03:26:49.824458 20629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:49.824733 20629 log.cc:824] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:49.825435 20629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b: No bootstrap required, opened a new log
I20260927 03:26:49.827173 20629 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e8977afd234ad79288e944d65a8c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39999 } }
I20260927 03:26:49.827322 20629 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:49.827343 20629 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e8977afd234ad79288e944d65a8c8b, State: Initialized, Role: FOLLOWER
I20260927 03:26:49.827472 20629 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e8977afd234ad79288e944d65a8c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39999 } }
I20260927 03:26:49.827540 20629 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:49.827562 20629 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:49.827618 20629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:49.827749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 20561
I20260927 03:26:49.827853 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:26:49.828022 392 external_mini_cluster.cc:1428] Setting key 46d1638bae42870f0c2fb06f5bf74b79
I20260927 03:26:49.828276 20629 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e8977afd234ad79288e944d65a8c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39999 } }
I20260927 03:26:49.828374 20629 leader_election.cc:302] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8e8977afd234ad79288e944d65a8c8b; no voters:
I20260927 03:26:49.828547 20629 leader_election.cc:288] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:26:49.828790 20629 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:49.828855 20635 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e8977afd234ad79288e944d65a8c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39999 } } }
I20260927 03:26:49.828959 20635 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:26:49.829157 20632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:49.829228 20632 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [term 1 LEADER]: Becoming Leader. State: Replica: e8e8977afd234ad79288e944d65a8c8b, State: Running, Role: LEADER
I20260927 03:26:49.829365 20632 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e8977afd234ad79288e944d65a8c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39999 } }
I20260927 03:26:49.829735 20635 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8e8977afd234ad79288e944d65a8c8b. Latest consensus state: current_term: 1 leader_uuid: "e8e8977afd234ad79288e944d65a8c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e8977afd234ad79288e944d65a8c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39999 } } }
I20260927 03:26:49.829815 20635 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b [sys.catalog]: This master's current role is: LEADER
W20260927 03:26:49.830457 20648 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:26:49.830513 20648 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:26:49.830555 20646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:49.830720 20646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:49.832252 20646 catalog_manager.cc:1347] Generated new cluster ID: c4621696f3fc403b897e8893bc851a66
I20260927 03:26:49.832305 20646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:26:49 dist-test-slave-gw2v krb5kdc[20548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 03:26:49.839689 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:49.828540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36193 (local address 127.0.0.1:39999)
0927 03:26:49.828713 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:49.828726 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:49.828765 (+ 39us) server_negotiation.cc:407] Connection header received
0927 03:26:49.829876 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:49.829884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:49.829945 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:49.830084 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:49.831219 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:49.832589 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:49.833350 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:49.833783 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:49.833878 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:49.836202 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:49.836207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:49.836207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:49.836218 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:49.838224 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:49.838580 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:49.838582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:49.838582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:49.838683 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:49.838900 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:49.838901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:49.838901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:49.839136 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:49.839302 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:49.839393 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:26:49.839493 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260927 03:26:49.840629 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:26:49.847029 20646 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:26:49.847597 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:49.841864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34553 (local address 127.0.0.1:39999)
0927 03:26:49.841898 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:49.841900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:49.842003 (+ 103us) server_negotiation.cc:407] Connection header received
0927 03:26:49.842047 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:49.842048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:49.842058 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:49.842100 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:49.842551 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:49.843103 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:49.843798 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:49.844022 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:49.844075 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:49.845197 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:49.845199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:49.845199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:49.845206 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:49.846410 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:49.846769 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:49.846769 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:49.846770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:49.846844 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:49.847106 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:49.847107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:49.847107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:49.847234 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:49.847337 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:49.847456 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 03:26:49.847515 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:26:49.847721 20646 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:49.856396 20646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8e8977afd234ad79288e944d65a8c8b: Generated new TSK 0
I20260927 03:26:49.856611 20646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:26:49.858897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 20561
W20260927 03:26:49.862475 20546 connection.cc:535] client connection to 127.0.0.1:39999 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39999): Connection reset by peer (error 104)
2026-09-27T03:26:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1231 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:26:49 dist-test-slave-gw2v krb5kdc[20659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:26:49 dist-test-slave-gw2v krb5kdc[20659](info): set up 2 sockets
Sep 27 03:26:49 dist-test-slave-gw2v krb5kdc[20659](info): commencing operation
krb5kdc: starting...
W20260927 03:26:50.810626 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:26:50 dist-test-slave-gw2v krb5kdc[20659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:26:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:26:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:26:50.929867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:51.009150 20672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:51.009388 20672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:51.009421 20672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:51.010632 20672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:51.010684 20672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:51.010701 20672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:51.010717 20672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:51.010731 20672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:51.012264 20672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33097
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:51.012532 20672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:51.012754 20672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:51.017786 20680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:51.017832 20678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:51.017799 20677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:51.017843 20672 server_base.cc:1034] running on GCE node
I20260927 03:26:51.018257 20672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:51.018532 20672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:51.019691 20672 hybrid_clock.cc:648] HybridClock initialized: now 1790479611019678 us; error 30 us; skew 500 ppm
Sep 27 03:26:51 dist-test-slave-gw2v krb5kdc[20659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:51.022205 20672 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:26:51.022615 20672 webserver.cc:460] Webserver started at http://127.0.0.1:37179/ using document root <none> and password file <none>
I20260927 03:26:51.022852 20672 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:51.022903 20672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:51.023023 20672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:51.024142 20672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "b1f5f9203e3a4eb78e31ceab4a9d004b"
format_stamp: "Formatted at 2026-09-27 03:26:51 on dist-test-slave-gw2v"
server_key: "baa4da4311ae772a747421e480457f6c"
server_key_iv: "5ef849accb352700b9e04105d484580c"
server_key_version: "encryptionkey@0"
I20260927 03:26:51.024479 20672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "b1f5f9203e3a4eb78e31ceab4a9d004b"
format_stamp: "Formatted at 2026-09-27 03:26:51 on dist-test-slave-gw2v"
server_key: "baa4da4311ae772a747421e480457f6c"
server_key_iv: "5ef849accb352700b9e04105d484580c"
server_key_version: "encryptionkey@0"
I20260927 03:26:51.025825 20672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:26:51.026635 20687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:51.026902 20672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:51.027004 20672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "b1f5f9203e3a4eb78e31ceab4a9d004b"
format_stamp: "Formatted at 2026-09-27 03:26:51 on dist-test-slave-gw2v"
server_key: "baa4da4311ae772a747421e480457f6c"
server_key_iv: "5ef849accb352700b9e04105d484580c"
server_key_version: "encryptionkey@0"
I20260927 03:26:51.027083 20672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:51.042439 20672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:51.044775 20672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:51.044899 20672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:51.049506 20672 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33097
I20260927 03:26:51.049543 20739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33097 every 8 connection(s)
I20260927 03:26:51.050016 20672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:51.050904 20740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:51.053274 20740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b: Bootstrap starting.
I20260927 03:26:51.053834 20740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:51.054106 20740 log.cc:824] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:51.054551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 20672
I20260927 03:26:51.054653 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:26:51.054808 392 external_mini_cluster.cc:1428] Setting key 908ef0693b845d005e5e0bceaa6f5546
I20260927 03:26:51.054920 20740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b: No bootstrap required, opened a new log
I20260927 03:26:51.057229 20740 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5f9203e3a4eb78e31ceab4a9d004b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33097 } }
I20260927 03:26:51.057374 20740 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:51.057397 20740 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f5f9203e3a4eb78e31ceab4a9d004b, State: Initialized, Role: FOLLOWER
I20260927 03:26:51.057511 20740 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5f9203e3a4eb78e31ceab4a9d004b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33097 } }
I20260927 03:26:51.057586 20740 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:51.057617 20740 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:51.057667 20740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:51.058233 20740 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5f9203e3a4eb78e31ceab4a9d004b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33097 } }
I20260927 03:26:51.058323 20740 leader_election.cc:302] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1f5f9203e3a4eb78e31ceab4a9d004b; no voters:
I20260927 03:26:51.058501 20740 leader_election.cc:288] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:26:51.058534 20745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:51.058696 20745 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [term 1 LEADER]: Becoming Leader. State: Replica: b1f5f9203e3a4eb78e31ceab4a9d004b, State: Running, Role: LEADER
I20260927 03:26:51.058756 20740 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:51.058790 20745 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5f9203e3a4eb78e31ceab4a9d004b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33097 } }
I20260927 03:26:51.059154 20745 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1f5f9203e3a4eb78e31ceab4a9d004b. Latest consensus state: current_term: 1 leader_uuid: "b1f5f9203e3a4eb78e31ceab4a9d004b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5f9203e3a4eb78e31ceab4a9d004b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33097 } } }
I20260927 03:26:51.059223 20745 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:51.059427 20752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:51.059541 20747 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1f5f9203e3a4eb78e31ceab4a9d004b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5f9203e3a4eb78e31ceab4a9d004b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33097 } } }
I20260927 03:26:51.059873 20747 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:51.060376 20752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:51.060596 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:51.055349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50671 (local address 127.0.0.1:33097)
0927 03:26:51.055519 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:51.055527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:51.055552 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:26:51.056132 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:51.056140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:51.056188 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:51.056328 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:51.056558 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:51.057528 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:51.058362 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:51.059037 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:51.059160 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:51.059884 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:51.059887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:51.059888 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:51.060101 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:51.060136 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:51.060149 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:51.060233 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:51.060368 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 03:26:51.060398 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260927 03:26:51.061685 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:26:51.066318 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:51.063230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52389 (local address 127.0.0.1:33097)
0927 03:26:51.063273 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:51.063275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:51.063342 (+ 67us) server_negotiation.cc:407] Connection header received
0927 03:26:51.063397 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:51.063398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:51.063407 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:51.063472 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:51.063601 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:51.064312 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:51.065051 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:51.065357 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:51.065426 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:51.066084 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:51.066085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:51.066085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:51.066089 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:51.066103 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:51.066108 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:51.066168 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:51.066226 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 03:26:51.066236 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:26:51.067263 20752 catalog_manager.cc:1347] Generated new cluster ID: ba76b8f9bf7c4ef5a8a9bd1d48d2236b
I20260927 03:26:51.067314 20752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:26:51.087071 20752 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:26:51.087725 20752 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:51.097146 20752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1f5f9203e3a4eb78e31ceab4a9d004b: Generated new TSK 0
I20260927 03:26:51.097352 20752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:26:51.101099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 20672
W20260927 03:26:51.105275 20657 connection.cc:535] client connection to 127.0.0.1:33097 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33097): Connection reset by peer (error 104)
2026-09-27T03:26:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1242 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:26:51 dist-test-slave-gw2v krb5kdc[20771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:26:51 dist-test-slave-gw2v krb5kdc[20771](info): set up 2 sockets
Sep 27 03:26:51 dist-test-slave-gw2v krb5kdc[20771](info): commencing operation
krb5kdc: starting...
W20260927 03:26:52.054570 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:26:52 dist-test-slave-gw2v krb5kdc[20771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:26:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:26:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:26:52.167627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:52.246591 20784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:52.246840 20784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:52.246876 20784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:52.248049 20784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:52.248090 20784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:52.248107 20784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:52.248123 20784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:52.248137 20784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:52.249619 20784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37867
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:52.249863 20784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:52.250070 20784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:52.254711 20789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:52.254739 20792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:52.254779 20790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:52.254840 20784 server_base.cc:1034] running on GCE node
I20260927 03:26:52.255134 20784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:52.255424 20784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:52.256587 20784 hybrid_clock.cc:648] HybridClock initialized: now 1790479612256570 us; error 27 us; skew 500 ppm
Sep 27 03:26:52 dist-test-slave-gw2v krb5kdc[20771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:52.259080 20784 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:26:52.259514 20784 webserver.cc:460] Webserver started at http://127.0.0.1:37175/ using document root <none> and password file <none>
I20260927 03:26:52.259704 20784 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:52.259752 20784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:52.259841 20784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:52.260861 20784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "82602064e5d24e1bae1e836c426156cc"
format_stamp: "Formatted at 2026-09-27 03:26:52 on dist-test-slave-gw2v"
server_key: "a636210f432148c7c4241d147990a5ed"
server_key_iv: "2d25bae824c3be7c43f0ceaa813fce4b"
server_key_version: "encryptionkey@0"
I20260927 03:26:52.261143 20784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "82602064e5d24e1bae1e836c426156cc"
format_stamp: "Formatted at 2026-09-27 03:26:52 on dist-test-slave-gw2v"
server_key: "a636210f432148c7c4241d147990a5ed"
server_key_iv: "2d25bae824c3be7c43f0ceaa813fce4b"
server_key_version: "encryptionkey@0"
I20260927 03:26:52.262370 20784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:26:52.263155 20799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:52.263330 20784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:52.263419 20784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "82602064e5d24e1bae1e836c426156cc"
format_stamp: "Formatted at 2026-09-27 03:26:52 on dist-test-slave-gw2v"
server_key: "a636210f432148c7c4241d147990a5ed"
server_key_iv: "2d25bae824c3be7c43f0ceaa813fce4b"
server_key_version: "encryptionkey@0"
I20260927 03:26:52.263482 20784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:52.266697 20784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:52.268433 20784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:52.268559 20784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:52.272328 20784 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37867
I20260927 03:26:52.272360 20851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37867 every 8 connection(s)
I20260927 03:26:52.272670 20784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:52.273571 20852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:52.275882 20852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc: Bootstrap starting.
I20260927 03:26:52.276441 20852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:52.276690 20852 log.cc:824] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:52.277416 20852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc: No bootstrap required, opened a new log
I20260927 03:26:52.279182 20852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82602064e5d24e1bae1e836c426156cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37867 } }
I20260927 03:26:52.279356 20852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:52.279395 20852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82602064e5d24e1bae1e836c426156cc, State: Initialized, Role: FOLLOWER
I20260927 03:26:52.279527 20852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82602064e5d24e1bae1e836c426156cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37867 } }
I20260927 03:26:52.279593 20852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:52.279623 20852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:52.279670 20852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:52.280202 20852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82602064e5d24e1bae1e836c426156cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37867 } }
I20260927 03:26:52.280295 20852 leader_election.cc:302] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82602064e5d24e1bae1e836c426156cc; no voters:
I20260927 03:26:52.280450 20852 leader_election.cc:288] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:26:52.280517 20855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:52.280700 20852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:52.280867 20855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [term 1 LEADER]: Becoming Leader. State: Replica: 82602064e5d24e1bae1e836c426156cc, State: Running, Role: LEADER
I20260927 03:26:52.280964 20855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82602064e5d24e1bae1e836c426156cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37867 } }
I20260927 03:26:52.281283 20856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82602064e5d24e1bae1e836c426156cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82602064e5d24e1bae1e836c426156cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37867 } } }
I20260927 03:26:52.281291 20857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82602064e5d24e1bae1e836c426156cc. Latest consensus state: current_term: 1 leader_uuid: "82602064e5d24e1bae1e836c426156cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82602064e5d24e1bae1e836c426156cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37867 } } }
I20260927 03:26:52.281376 20856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:52.281392 20857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:52.281566 20864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:52.282069 20864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:52.282176 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 20784
I20260927 03:26:52.282305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:26:52.282487 392 external_mini_cluster.cc:1428] Setting key 8c1c0b25690b62edee0e373e53ba8fc7
I20260927 03:26:52.283740 20864 catalog_manager.cc:1347] Generated new cluster ID: 0cefb384f55d4650809e04355f77bce4
I20260927 03:26:52.283787 20864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:26:52.284886 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:52.282973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59639 (local address 127.0.0.1:37867)
0927 03:26:52.283384 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:52.283394 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:52.283426 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:26:52.283540 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:52.283547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:52.283579 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:52.283720 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:52.284487 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:52.284490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:52.284491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:52.284495 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:52.284535 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:52.284548 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:52.284603 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:52.284674 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 03:26:52.284712 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":81,"threads_started":1}
I20260927 03:26:52.288043 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:26:52.292279 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:52.291145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50375 (local address 127.0.0.1:37867)
0927 03:26:52.291184 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:52.291186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:52.291345 (+ 159us) server_negotiation.cc:407] Connection header received
0927 03:26:52.291396 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:52.291397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:52.291404 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:52.291456 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:52.292073 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:52.292074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:52.292075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:52.292078 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:52.292095 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:52.292100 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:52.292151 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:52.292200 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 03:26:52.292213 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:26:52.293279 20864 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:26:52.293421 20864 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:52.304622 20864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82602064e5d24e1bae1e836c426156cc: Generated new TSK 0
I20260927 03:26:52.304785 20864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:26:52.307511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 20784
2026-09-27T03:26:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:26:52 dist-test-slave-gw2v krb5kdc[20883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:26:52 dist-test-slave-gw2v krb5kdc[20883](info): set up 2 sockets
Sep 27 03:26:52 dist-test-slave-gw2v krb5kdc[20883](info): commencing operation
krb5kdc: starting...
W20260927 03:26:53.254667 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:26:53 dist-test-slave-gw2v krb5kdc[20883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:26:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:26:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:26:53.372076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:53.452811 20896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:53.453029 20896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:53.453052 20896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:53.454207 20896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:53.454250 20896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:53.454264 20896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:53.454275 20896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:53.454286 20896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:53.455756 20896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:53.455952 20896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:53.456143 20896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:53.461063 20904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:53.461191 20896 server_base.cc:1034] running on GCE node
W20260927 03:26:53.461108 20901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:53.461284 20902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:53.461552 20896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:53.461871 20896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:53.463024 20896 hybrid_clock.cc:648] HybridClock initialized: now 1790479613463001 us; error 31 us; skew 500 ppm
Sep 27 03:26:53 dist-test-slave-gw2v krb5kdc[20883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:53.470203 20896 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:26:53.470733 20896 webserver.cc:460] Webserver started at http://127.0.0.1:41371/ using document root <none> and password file <none>
I20260927 03:26:53.471021 20896 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:53.471079 20896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:53.471217 20896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:53.472373 20896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "2ca9703b312745a2a0f11f900c776977"
format_stamp: "Formatted at 2026-09-27 03:26:53 on dist-test-slave-gw2v"
server_key: "23c5c7b2feffdccee1b992aa31e35166"
server_key_iv: "29c70338215def0470a644c310dcfe94"
server_key_version: "encryptionkey@0"
I20260927 03:26:53.472723 20896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ca9703b312745a2a0f11f900c776977"
format_stamp: "Formatted at 2026-09-27 03:26:53 on dist-test-slave-gw2v"
server_key: "23c5c7b2feffdccee1b992aa31e35166"
server_key_iv: "29c70338215def0470a644c310dcfe94"
server_key_version: "encryptionkey@0"
I20260927 03:26:53.474187 20896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:26:53.474946 20911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:53.475167 20896 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:53.475262 20896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "2ca9703b312745a2a0f11f900c776977"
format_stamp: "Formatted at 2026-09-27 03:26:53 on dist-test-slave-gw2v"
server_key: "23c5c7b2feffdccee1b992aa31e35166"
server_key_iv: "29c70338215def0470a644c310dcfe94"
server_key_version: "encryptionkey@0"
I20260927 03:26:53.475337 20896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:53.487308 20896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:53.489367 20896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:53.489497 20896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:53.493219 20896 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38005
I20260927 03:26:53.493266 20963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38005 every 8 connection(s)
I20260927 03:26:53.493577 20896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:53.494371 20964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:53.496667 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977: Bootstrap starting.
I20260927 03:26:53.496903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 20896
I20260927 03:26:53.497007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:26:53.497148 392 external_mini_cluster.cc:1428] Setting key 09efed98d4d5f6e4cb93b8801bc97b4c
I20260927 03:26:53.497304 20964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:53.497635 20964 log.cc:824] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:53.498418 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977: No bootstrap required, opened a new log
I20260927 03:26:53.500680 20964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9703b312745a2a0f11f900c776977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260927 03:26:53.500854 20964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:53.500880 20964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca9703b312745a2a0f11f900c776977, State: Initialized, Role: FOLLOWER
I20260927 03:26:53.501003 20964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9703b312745a2a0f11f900c776977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260927 03:26:53.501077 20964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:53.501101 20964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:53.501152 20964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:53.501762 20964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9703b312745a2a0f11f900c776977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260927 03:26:53.501879 20964 leader_election.cc:302] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ca9703b312745a2a0f11f900c776977; no voters:
I20260927 03:26:53.502035 20964 leader_election.cc:288] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:26:53.502076 20969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:53.502228 20969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [term 1 LEADER]: Becoming Leader. State: Replica: 2ca9703b312745a2a0f11f900c776977, State: Running, Role: LEADER
I20260927 03:26:53.502283 20964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:53.502400 20969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9703b312745a2a0f11f900c776977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260927 03:26:53.503051 20970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ca9703b312745a2a0f11f900c776977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9703b312745a2a0f11f900c776977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } } }
I20260927 03:26:53.503154 20970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:53.503052 20971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ca9703b312745a2a0f11f900c776977. Latest consensus state: current_term: 1 leader_uuid: "2ca9703b312745a2a0f11f900c776977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9703b312745a2a0f11f900c776977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } } }
I20260927 03:26:53.503233 20971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977 [sys.catalog]: This master's current role is: LEADER
W20260927 03:26:53.504288 20984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
Sep 27 03:26:53 dist-test-slave-gw2v krb5kdc[20883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260927 03:26:53.504343 20984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:26:53.504405 20978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:53.504607 20978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:53.506188 20978 catalog_manager.cc:1347] Generated new cluster ID: 00cf3f9532c44088b9179ac1d3bd68ea
I20260927 03:26:53.506240 20978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:26:53.508579 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:53.497649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56307 (local address 127.0.0.1:38005)
0927 03:26:53.497845 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:53.497855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:53.497887 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:26:53.498400 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:53.498409 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:53.498471 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:53.498625 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:53.499238 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:53.500633 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:53.501522 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:53.502123 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:53.502550 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:53.504800 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:53.504805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:53.504805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:53.504830 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:53.506760 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:53.507291 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:53.507295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:53.507295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:53.507430 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:53.507611 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:53.507612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:53.507612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:53.507938 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:53.508104 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:53.508203 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 03:26:53.508350 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
I20260927 03:26:53.509460 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:26:53.521894 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:53.511029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50523 (local address 127.0.0.1:38005)
0927 03:26:53.511065 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:53.511066 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:53.515047 (+ 3981us) server_negotiation.cc:407] Connection header received
0927 03:26:53.515103 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:53.515104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:53.515115 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:53.515176 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:53.515827 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:53.516585 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:53.517595 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:53.517901 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:53.517953 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:53.519320 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:53.519322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:53.519322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:53.519328 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:53.520809 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:53.521177 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:53.521177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:53.521178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:53.521230 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:53.521417 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:53.521417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:53.521418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:53.521543 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:53.521634 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:53.521732 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 03:26:53.521802 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:26:53.522778 20978 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:26:53.523276 20978 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:53.528038 20978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ca9703b312745a2a0f11f900c776977: Generated new TSK 0
I20260927 03:26:53.528261 20978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:26:53.532596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 20896
2026-09-27T03:26:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:26:53 dist-test-slave-gw2v krb5kdc[20995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:26:53 dist-test-slave-gw2v krb5kdc[20995](info): set up 2 sockets
Sep 27 03:26:53 dist-test-slave-gw2v krb5kdc[20995](info): commencing operation
krb5kdc: starting...
W20260927 03:26:54.490582 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:26:54 dist-test-slave-gw2v krb5kdc[20995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:26:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:26:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:26:54.609078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:54.689989 21008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:54.690205 21008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:54.690227 21008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:54.691452 21008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:54.691494 21008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:54.691507 21008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:54.691519 21008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:54.691529 21008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:54.692989 21008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:54.693219 21008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:54.693436 21008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:54.698230 21014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:54.698235 21016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:54.698235 21013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:54.698536 21008 server_base.cc:1034] running on GCE node
I20260927 03:26:54.698760 21008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:54.699065 21008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:54.700203 21008 hybrid_clock.cc:648] HybridClock initialized: now 1790479614700193 us; error 26 us; skew 500 ppm
Sep 27 03:26:54 dist-test-slave-gw2v krb5kdc[20995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:54.702716 21008 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:26:54.703177 21008 webserver.cc:460] Webserver started at http://127.0.0.1:37561/ using document root <none> and password file <none>
I20260927 03:26:54.703377 21008 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:54.703433 21008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:54.703552 21008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:54.704581 21008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "7d3c24bd03544d8f84b818e239f4f2b8"
format_stamp: "Formatted at 2026-09-27 03:26:54 on dist-test-slave-gw2v"
server_key: "2f5a4385ea4513f8db7b4e13c28b4928"
server_key_iv: "fbd0f69494bdf566915d2f39d30d0ea9"
server_key_version: "encryptionkey@0"
I20260927 03:26:54.704900 21008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d3c24bd03544d8f84b818e239f4f2b8"
format_stamp: "Formatted at 2026-09-27 03:26:54 on dist-test-slave-gw2v"
server_key: "2f5a4385ea4513f8db7b4e13c28b4928"
server_key_iv: "fbd0f69494bdf566915d2f39d30d0ea9"
server_key_version: "encryptionkey@0"
I20260927 03:26:54.706131 21008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:26:54.706914 21023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:54.707077 21008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:54.707158 21008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "7d3c24bd03544d8f84b818e239f4f2b8"
format_stamp: "Formatted at 2026-09-27 03:26:54 on dist-test-slave-gw2v"
server_key: "2f5a4385ea4513f8db7b4e13c28b4928"
server_key_iv: "fbd0f69494bdf566915d2f39d30d0ea9"
server_key_version: "encryptionkey@0"
I20260927 03:26:54.707226 21008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:54.753545 21008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:54.755805 21008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:54.755956 21008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:54.759656 21008 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40237
I20260927 03:26:54.759703 21075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40237 every 8 connection(s)
I20260927 03:26:54.760038 21008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:54.760843 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:54.763154 21076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8: Bootstrap starting.
I20260927 03:26:54.763684 21076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:54.763907 21076 log.cc:824] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:54.764484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 21008
I20260927 03:26:54.764608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:26:54.764639 21076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8: No bootstrap required, opened a new log
I20260927 03:26:54.764789 392 external_mini_cluster.cc:1428] Setting key 057069afc06f39d2f1516439e8a16302
I20260927 03:26:54.766355 21076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3c24bd03544d8f84b818e239f4f2b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260927 03:26:54.766506 21076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:54.766528 21076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d3c24bd03544d8f84b818e239f4f2b8, State: Initialized, Role: FOLLOWER
I20260927 03:26:54.766645 21076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3c24bd03544d8f84b818e239f4f2b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260927 03:26:54.766726 21076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:54.766764 21076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:54.766810 21076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:54.767916 21076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3c24bd03544d8f84b818e239f4f2b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260927 03:26:54.768028 21076 leader_election.cc:302] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d3c24bd03544d8f84b818e239f4f2b8; no voters:
I20260927 03:26:54.768180 21076 leader_election.cc:288] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:26:54.768213 21081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:54.768363 21081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7d3c24bd03544d8f84b818e239f4f2b8, State: Running, Role: LEADER
I20260927 03:26:54.768417 21076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:54.768613 21081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3c24bd03544d8f84b818e239f4f2b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260927 03:26:54.769033 21083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d3c24bd03544d8f84b818e239f4f2b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3c24bd03544d8f84b818e239f4f2b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } } }
I20260927 03:26:54.769114 21083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:54.769268 21082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d3c24bd03544d8f84b818e239f4f2b8. Latest consensus state: current_term: 1 leader_uuid: "7d3c24bd03544d8f84b818e239f4f2b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3c24bd03544d8f84b818e239f4f2b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } } }
I20260927 03:26:54.769316 21082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8 [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:54.769604 21090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:54.770107 21090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:54.771726 21090 catalog_manager.cc:1347] Generated new cluster ID: ac96216183734d46bbaa4d43b240f20a
I20260927 03:26:54.771772 21090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:26:54.772375 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:54.765319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59947 (local address 127.0.0.1:40237)
0927 03:26:54.765484 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:54.765492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:54.766868 (+ 1376us) server_negotiation.cc:407] Connection header received
0927 03:26:54.767086 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:54.767094 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:54.767152 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:54.767293 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:54.767519 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:54.768835 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:54.769953 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:54.770597 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:54.770707 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:54.771462 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:54.771466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:54.771466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:54.771682 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:54.771720 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:54.771736 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:54.771799 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:54.772083 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 03:26:54.772151 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260927 03:26:54.773326 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:26:54.782652 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:54.774816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34633 (local address 127.0.0.1:40237)
0927 03:26:54.774873 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:54.774874 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:54.779109 (+ 4235us) server_negotiation.cc:407] Connection header received
0927 03:26:54.779162 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:54.779163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:54.779175 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:54.779228 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:54.779348 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:54.780094 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:54.781067 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:54.781364 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:54.781425 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:54.782075 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:54.782077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:54.782077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:54.782082 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:54.782127 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:54.782132 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:54.782182 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:54.782479 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 03:26:54.782507 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:26:54.788264 21090 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:26:54.788879 21090 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:54.806165 21090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d3c24bd03544d8f84b818e239f4f2b8: Generated new TSK 0
I20260927 03:26:54.806384 21090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:26:54.812973 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 21008
2026-09-27T03:26:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:26:54 dist-test-slave-gw2v krb5kdc[21107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:26:54 dist-test-slave-gw2v krb5kdc[21107](info): set up 2 sockets
Sep 27 03:26:54 dist-test-slave-gw2v krb5kdc[21107](info): commencing operation
krb5kdc: starting...
W20260927 03:26:55.762740 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:26:55 dist-test-slave-gw2v krb5kdc[21107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:26:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:26:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:26:55.882402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:55.962100 21120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:55.962317 21120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:55.962349 21120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:55.963537 21120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:55.963578 21120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:55.963595 21120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:55.963611 21120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:55.963625 21120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:55.965088 21120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:55.965332 21120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:55.965538 21120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:55.970280 21128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:55.970232 21125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:55.970494 21120 server_base.cc:1034] running on GCE node
W20260927 03:26:55.970314 21126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:55.970808 21120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:55.971161 21120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:55.972323 21120 hybrid_clock.cc:648] HybridClock initialized: now 1790479615972300 us; error 34 us; skew 500 ppm
Sep 27 03:26:55 dist-test-slave-gw2v krb5kdc[21107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:55.974879 21120 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:26:55.975307 21120 webserver.cc:460] Webserver started at http://127.0.0.1:44609/ using document root <none> and password file <none>
I20260927 03:26:55.975497 21120 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:55.975543 21120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:55.975636 21120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:55.976697 21120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "c6113a165c5f48eb9a5205df3c0dffbe"
format_stamp: "Formatted at 2026-09-27 03:26:55 on dist-test-slave-gw2v"
server_key: "9f465086510341c5fce47d300455b1a4"
server_key_iv: "4360dca7b2eef681772a18a9f1dce531"
server_key_version: "encryptionkey@0"
I20260927 03:26:55.976995 21120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "c6113a165c5f48eb9a5205df3c0dffbe"
format_stamp: "Formatted at 2026-09-27 03:26:55 on dist-test-slave-gw2v"
server_key: "9f465086510341c5fce47d300455b1a4"
server_key_iv: "4360dca7b2eef681772a18a9f1dce531"
server_key_version: "encryptionkey@0"
I20260927 03:26:55.978219 21120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:26:55.978945 21135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:55.979157 21120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:26:55.979275 21120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "c6113a165c5f48eb9a5205df3c0dffbe"
format_stamp: "Formatted at 2026-09-27 03:26:55 on dist-test-slave-gw2v"
server_key: "9f465086510341c5fce47d300455b1a4"
server_key_iv: "4360dca7b2eef681772a18a9f1dce531"
server_key_version: "encryptionkey@0"
I20260927 03:26:55.979342 21120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:55.982933 21120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:55.985188 21120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:55.985319 21120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:55.989030 21120 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46307
I20260927 03:26:55.989130 21187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46307 every 8 connection(s)
I20260927 03:26:55.989372 21120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:55.990283 21188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:55.992559 21188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe: Bootstrap starting.
I20260927 03:26:55.993104 21188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:55.993346 21188 log.cc:824] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:55.993952 21188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe: No bootstrap required, opened a new log
I20260927 03:26:55.995878 21188 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6113a165c5f48eb9a5205df3c0dffbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260927 03:26:55.996033 21188 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:55.996055 21188 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6113a165c5f48eb9a5205df3c0dffbe, State: Initialized, Role: FOLLOWER
I20260927 03:26:55.996169 21188 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6113a165c5f48eb9a5205df3c0dffbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260927 03:26:55.996243 21188 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:55.996270 21188 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:55.996304 21188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:55.996862 21188 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6113a165c5f48eb9a5205df3c0dffbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260927 03:26:55.996942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 21120
I20260927 03:26:55.996973 21188 leader_election.cc:302] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6113a165c5f48eb9a5205df3c0dffbe; no voters:
I20260927 03:26:55.997046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:26:55.997162 21188 leader_election.cc:288] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:26:55.997227 21191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:55.997255 392 external_mini_cluster.cc:1428] Setting key b56c7aac7b296befd6ce571a2e7f9b8e
I20260927 03:26:55.997424 21188 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:55.997442 21191 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [term 1 LEADER]: Becoming Leader. State: Replica: c6113a165c5f48eb9a5205df3c0dffbe, State: Running, Role: LEADER
I20260927 03:26:55.997622 21191 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6113a165c5f48eb9a5205df3c0dffbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260927 03:26:55.998206 21192 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6113a165c5f48eb9a5205df3c0dffbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6113a165c5f48eb9a5205df3c0dffbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } } }
I20260927 03:26:55.998292 21192 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:55.998471 21193 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6113a165c5f48eb9a5205df3c0dffbe. Latest consensus state: current_term: 1 leader_uuid: "c6113a165c5f48eb9a5205df3c0dffbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6113a165c5f48eb9a5205df3c0dffbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } } }
I20260927 03:26:55.998530 21193 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:55.998775 21202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:55.999435 21202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:55.999740 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:55.997803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43941 (local address 127.0.0.1:46307)
0927 03:26:55.997972 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:55.997984 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:55.998020 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:26:55.998141 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:55.998148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:55.998183 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:55.998651 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:55.999219 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:55.999223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:55.999223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:55.999227 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:55.999272 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:55.999287 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:55.999348 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:55.999489 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 03:26:55.999530 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260927 03:26:56.000790 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:26:56.001267 21202 catalog_manager.cc:1347] Generated new cluster ID: 2b5bd876189e435f8616d28e2ab394df
I20260927 03:26:56.001322 21202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:26:56.004230 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:56.003164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43193 (local address 127.0.0.1:46307)
0927 03:26:56.003206 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:56.003208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:56.003353 (+ 145us) server_negotiation.cc:407] Connection header received
0927 03:26:56.003408 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:56.003409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:56.003414 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:56.003466 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:56.004018 (+ 552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:56.004019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:56.004020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:56.004024 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:56.004040 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:56.004045 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:56.004094 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:56.004141 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 03:26:56.004153 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:26:56.013185 21202 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:26:56.013383 21202 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:56.022485 21202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6113a165c5f48eb9a5205df3c0dffbe: Generated new TSK 0
I20260927 03:26:56.022751 21202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:26:56.025312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 21120
2026-09-27T03:26:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:26:56 dist-test-slave-gw2v krb5kdc[21219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:26:56 dist-test-slave-gw2v krb5kdc[21219](info): set up 2 sockets
Sep 27 03:26:56 dist-test-slave-gw2v krb5kdc[21219](info): commencing operation
krb5kdc: starting...
W20260927 03:26:56.974745 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:26:57 dist-test-slave-gw2v krb5kdc[21219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:26:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:26:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:26:57.090485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:57.170334 21232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:57.170547 21232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:57.170578 21232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:57.171767 21232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:57.171824 21232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:57.171844 21232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:57.171859 21232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:57.171873 21232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:57.173314 21232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:57.173552 21232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:57.173756 21232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:57.178592 21237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:57.178598 21238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:57.178597 21240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:57.178975 21232 server_base.cc:1034] running on GCE node
I20260927 03:26:57.179221 21232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:57.179520 21232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:57.180682 21232 hybrid_clock.cc:648] HybridClock initialized: now 1790479617180657 us; error 38 us; skew 500 ppm
Sep 27 03:26:57 dist-test-slave-gw2v krb5kdc[21219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:57.183233 21232 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:26:57.183729 21232 webserver.cc:460] Webserver started at http://127.0.0.1:44485/ using document root <none> and password file <none>
I20260927 03:26:57.183935 21232 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:57.183983 21232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:57.184109 21232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:57.185173 21232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "00be458becca4719bb0fd61c31749e6b"
format_stamp: "Formatted at 2026-09-27 03:26:57 on dist-test-slave-gw2v"
server_key: "f0004a5705cd65643aba793e0bceb7e6"
server_key_iv: "333cd8e582a30e5a25b8b30d77bdc061"
server_key_version: "encryptionkey@0"
I20260927 03:26:57.185513 21232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "00be458becca4719bb0fd61c31749e6b"
format_stamp: "Formatted at 2026-09-27 03:26:57 on dist-test-slave-gw2v"
server_key: "f0004a5705cd65643aba793e0bceb7e6"
server_key_iv: "333cd8e582a30e5a25b8b30d77bdc061"
server_key_version: "encryptionkey@0"
I20260927 03:26:57.186743 21232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:26:57.187469 21247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:57.187628 21232 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:57.187711 21232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "00be458becca4719bb0fd61c31749e6b"
format_stamp: "Formatted at 2026-09-27 03:26:57 on dist-test-slave-gw2v"
server_key: "f0004a5705cd65643aba793e0bceb7e6"
server_key_iv: "333cd8e582a30e5a25b8b30d77bdc061"
server_key_version: "encryptionkey@0"
I20260927 03:26:57.187778 21232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:57.221980 21232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:57.224247 21232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:57.224393 21232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:57.228278 21232 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44685
I20260927 03:26:57.228353 21299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44685 every 8 connection(s)
I20260927 03:26:57.228667 21232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:57.229530 21300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:57.231990 21300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b: Bootstrap starting.
I20260927 03:26:57.232549 21300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:57.232820 21300 log.cc:824] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:57.233546 21300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b: No bootstrap required, opened a new log
I20260927 03:26:57.235275 21300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00be458becca4719bb0fd61c31749e6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44685 } }
I20260927 03:26:57.235431 21300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:57.235466 21300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00be458becca4719bb0fd61c31749e6b, State: Initialized, Role: FOLLOWER
I20260927 03:26:57.235596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 21232
I20260927 03:26:57.235625 21300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00be458becca4719bb0fd61c31749e6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44685 } }
I20260927 03:26:57.235718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:26:57.235772 21300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:57.235826 21300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:57.235878 21300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:57.235886 392 external_mini_cluster.cc:1428] Setting key da2a607d2fe74f4e1090531421e49dcc
I20260927 03:26:57.236553 21300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00be458becca4719bb0fd61c31749e6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44685 } }
I20260927 03:26:57.236667 21300 leader_election.cc:302] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00be458becca4719bb0fd61c31749e6b; no voters:
I20260927 03:26:57.236835 21300 leader_election.cc:288] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:26:57.236862 21305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:57.237061 21305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [term 1 LEADER]: Becoming Leader. State: Replica: 00be458becca4719bb0fd61c31749e6b, State: Running, Role: LEADER
I20260927 03:26:57.237084 21300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:57.237219 21305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00be458becca4719bb0fd61c31749e6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44685 } }
I20260927 03:26:57.237707 21307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00be458becca4719bb0fd61c31749e6b. Latest consensus state: current_term: 1 leader_uuid: "00be458becca4719bb0fd61c31749e6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00be458becca4719bb0fd61c31749e6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44685 } } }
I20260927 03:26:57.237741 21306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00be458becca4719bb0fd61c31749e6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00be458becca4719bb0fd61c31749e6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44685 } } }
I20260927 03:26:57.237816 21306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:57.237804 21307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:57.238046 21314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:57.238601 21314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:57.240792 21314 catalog_manager.cc:1347] Generated new cluster ID: bf5853843b594c21901eb0ff5d6d5706
I20260927 03:26:57.240860 21314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:26:57 dist-test-slave-gw2v krb5kdc[21219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 03:26:57.247936 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:57.236651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:42761 (local address 127.0.0.1:44685)
0927 03:26:57.236836 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:57.236850 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:57.236888 (+ 38us) server_negotiation.cc:407] Connection header received
0927 03:26:57.237018 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:57.237026 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:57.237092 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:57.238109 (+ 1017us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:57.238250 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:57.239638 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:57.240728 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:57.241197 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:57.241297 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:57.243662 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:57.243666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:57.243667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:57.243680 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:57.245236 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:57.245599 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:57.245604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:57.245604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:57.245704 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:57.245905 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:57.245905 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:57.245906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:57.246168 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:57.246331 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:57.247678 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0927 03:26:57.247768 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260927 03:26:57.248817 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:26:57.255638 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:57.250075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:47933 (local address 127.0.0.1:44685)
0927 03:26:57.250110 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:57.250111 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:57.250205 (+ 94us) server_negotiation.cc:407] Connection header received
0927 03:26:57.250254 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:57.250255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:57.250265 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:57.250308 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:57.250788 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:57.251305 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:57.251984 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:57.252216 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:57.252271 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:57.253342 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:57.253344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:57.253345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:26:57.253351 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:57.254572 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:57.254926 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:57.254927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:57.254928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:57.254981 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:26:57.255177 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:26:57.255178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:26:57.255178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:26:57.255284 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:57.255385 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:57.255490 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 03:26:57.255547 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:26:57.272850 21314 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:26:57.273623 21314 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:57.281142 21314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00be458becca4719bb0fd61c31749e6b: Generated new TSK 0
I20260927 03:26:57.281368 21314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:26:57.288226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 21232
W20260927 03:26:57.292412 21217 connection.cc:535] client connection to 127.0.0.1:44685 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44685): Connection reset by peer (error 104)
2026-09-27T03:26:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1263 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:26:57 dist-test-slave-gw2v krb5kdc[21331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:26:57 dist-test-slave-gw2v krb5kdc[21331](info): set up 2 sockets
Sep 27 03:26:57 dist-test-slave-gw2v krb5kdc[21331](info): commencing operation
krb5kdc: starting...
W20260927 03:26:58.258560 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:26:58 dist-test-slave-gw2v krb5kdc[21331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:26:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:26:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:26:58.378543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39153
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:58.460283 21344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:58.460500 21344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:58.460520 21344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:58.461680 21344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:58.461728 21344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:58.461748 21344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:58.461760 21344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:58.461771 21344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:58.463359 21344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:58.463596 21344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:58.463805 21344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:58.468655 21350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:58.468647 21352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:58.468647 21349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:58.468690 21344 server_base.cc:1034] running on GCE node
I20260927 03:26:58.469040 21344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:58.469308 21344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:58.470489 21344 hybrid_clock.cc:648] HybridClock initialized: now 1790479618470471 us; error 29 us; skew 500 ppm
Sep 27 03:26:58 dist-test-slave-gw2v krb5kdc[21331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:58.476784 21344 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:26:58.477267 21344 webserver.cc:460] Webserver started at http://127.0.0.1:44509/ using document root <none> and password file <none>
I20260927 03:26:58.477461 21344 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:58.477504 21344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:58.477594 21344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:58.478595 21344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "dfbe8b9936c94e88bdd3e5803ae2e181"
format_stamp: "Formatted at 2026-09-27 03:26:58 on dist-test-slave-gw2v"
server_key: "054406ded29187d6faec80a6c95ae728"
server_key_iv: "c4fcbb551556c9083ffc825f99f9098f"
server_key_version: "encryptionkey@0"
I20260927 03:26:58.478928 21344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "dfbe8b9936c94e88bdd3e5803ae2e181"
format_stamp: "Formatted at 2026-09-27 03:26:58 on dist-test-slave-gw2v"
server_key: "054406ded29187d6faec80a6c95ae728"
server_key_iv: "c4fcbb551556c9083ffc825f99f9098f"
server_key_version: "encryptionkey@0"
I20260927 03:26:58.480152 21344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:58.480918 21359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:58.481091 21344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:58.481173 21344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "dfbe8b9936c94e88bdd3e5803ae2e181"
format_stamp: "Formatted at 2026-09-27 03:26:58 on dist-test-slave-gw2v"
server_key: "054406ded29187d6faec80a6c95ae728"
server_key_iv: "c4fcbb551556c9083ffc825f99f9098f"
server_key_version: "encryptionkey@0"
I20260927 03:26:58.481231 21344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:58.494073 21344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:58.496490 21344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:58.496615 21344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:58.500427 21344 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39153
I20260927 03:26:58.500463 21411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39153 every 8 connection(s)
I20260927 03:26:58.500788 21344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:58.501606 21412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:58.503691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 21344
I20260927 03:26:58.503793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:26:58.503957 392 external_mini_cluster.cc:1428] Setting key 2f6e2cf4f8bbadfcd0c6aa8ce370cd02
I20260927 03:26:58.503979 21412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181: Bootstrap starting.
I20260927 03:26:58.504647 21412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:58.504958 21412 log.cc:824] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:58.505668 21412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181: No bootstrap required, opened a new log
I20260927 03:26:58.507874 21412 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbe8b9936c94e88bdd3e5803ae2e181" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39153 } }
I20260927 03:26:58.508049 21412 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:58.508082 21412 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfbe8b9936c94e88bdd3e5803ae2e181, State: Initialized, Role: FOLLOWER
I20260927 03:26:58.508208 21412 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbe8b9936c94e88bdd3e5803ae2e181" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39153 } }
I20260927 03:26:58.508273 21412 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:58.508307 21412 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:58.508353 21412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:58.508997 21412 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbe8b9936c94e88bdd3e5803ae2e181" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39153 } }
I20260927 03:26:58.509097 21412 leader_election.cc:302] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfbe8b9936c94e88bdd3e5803ae2e181; no voters:
I20260927 03:26:58.509290 21412 leader_election.cc:288] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:26:58.509335 21417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:58.509508 21417 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [term 1 LEADER]: Becoming Leader. State: Replica: dfbe8b9936c94e88bdd3e5803ae2e181, State: Running, Role: LEADER
I20260927 03:26:58.509555 21412 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:58.509685 21417 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbe8b9936c94e88bdd3e5803ae2e181" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39153 } }
I20260927 03:26:58.510073 21418 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfbe8b9936c94e88bdd3e5803ae2e181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbe8b9936c94e88bdd3e5803ae2e181" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39153 } } }
I20260927 03:26:58.510146 21418 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:58.510159 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:58.504481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:41161 (local address 127.0.0.1:39153)
0927 03:26:58.504672 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:58.504681 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:58.504718 (+ 37us) server_negotiation.cc:407] Connection header received
0927 03:26:58.505273 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:58.505281 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:58.505353 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:58.505503 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:58.505698 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:58.507137 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:58.507943 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:58.508548 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:58.508655 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:58.509333 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:58.509337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:58.509337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:58.509560 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:58.509601 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:58.509616 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:58.509739 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:58.509917 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 03:26:58.509945 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260927 03:26:58.510789 21419 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfbe8b9936c94e88bdd3e5803ae2e181. Latest consensus state: current_term: 1 leader_uuid: "dfbe8b9936c94e88bdd3e5803ae2e181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbe8b9936c94e88bdd3e5803ae2e181" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39153 } } }
I20260927 03:26:58.510983 21419 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181 [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:58.510989 21426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:58.511236 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:26:58.511518 21426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:58.514127 21426 catalog_manager.cc:1347] Generated new cluster ID: 67448445e5564f67a3f800a074abc343
I20260927 03:26:58.514179 21426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:26:58.515609 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:58.512670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:60277 (local address 127.0.0.1:39153)
0927 03:26:58.512702 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:58.512703 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:58.512947 (+ 244us) server_negotiation.cc:407] Connection header received
0927 03:26:58.512994 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:58.512995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:58.513005 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:58.513055 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:58.513173 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:58.513710 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:58.514500 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:26:58.514708 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:26:58.514766 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:26:58.515387 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:58.515388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:58.515389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:58.515393 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:58.515407 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:58.515411 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:58.515458 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:58.515527 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 03:26:58.515537 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 03:26:58.541289 21426 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:26:58.542057 21426 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:58.549096 21426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfbe8b9936c94e88bdd3e5803ae2e181: Generated new TSK 0
I20260927 03:26:58.549332 21426 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:26:58.555971 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 21344
W20260927 03:26:58.559866 21329 connection.cc:535] client connection to 127.0.0.1:39153 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39153): Connection reset by peer (error 104)
2026-09-27T03:26:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1267 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:26:58 dist-test-slave-gw2v krb5kdc[21443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:26:58 dist-test-slave-gw2v krb5kdc[21443](info): set up 2 sockets
Sep 27 03:26:58 dist-test-slave-gw2v krb5kdc[21443](info): commencing operation
krb5kdc: starting...
W20260927 03:26:59.506798 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:26:59 dist-test-slave-gw2v krb5kdc[21443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:26:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:26:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:26:59.621254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:26:59.704833 21456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:26:59.705056 21456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:26:59.705087 21456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:26:59.706245 21456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:26:59.706287 21456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:26:59.706305 21456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:26:59.706322 21456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:26:59.706336 21456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:26:59.707768 21456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:26:59.708019 21456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:26:59.708220 21456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:26:59.712934 21462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:59.712971 21456 server_base.cc:1034] running on GCE node
W20260927 03:26:59.712945 21461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:26:59.712939 21464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:26:59.713379 21456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:26:59.713660 21456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:26:59.714841 21456 hybrid_clock.cc:648] HybridClock initialized: now 1790479619714797 us; error 31 us; skew 500 ppm
Sep 27 03:26:59 dist-test-slave-gw2v krb5kdc[21443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:26:59.717312 21456 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:26:59.717795 21456 webserver.cc:460] Webserver started at http://127.0.0.1:44573/ using document root <none> and password file <none>
I20260927 03:26:59.717991 21456 fs_manager.cc:359] Metadata directory not provided
I20260927 03:26:59.718039 21456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:26:59.718160 21456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:26:59.719260 21456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "53e5f6ac74314dbc880a0192d1c511c1"
format_stamp: "Formatted at 2026-09-27 03:26:59 on dist-test-slave-gw2v"
server_key: "3b45e4df364b58184057c81fabe06502"
server_key_iv: "20f2d56ac997d8abb9cebd2e88299a2f"
server_key_version: "encryptionkey@0"
I20260927 03:26:59.719597 21456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "53e5f6ac74314dbc880a0192d1c511c1"
format_stamp: "Formatted at 2026-09-27 03:26:59 on dist-test-slave-gw2v"
server_key: "3b45e4df364b58184057c81fabe06502"
server_key_iv: "20f2d56ac997d8abb9cebd2e88299a2f"
server_key_version: "encryptionkey@0"
I20260927 03:26:59.720849 21456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:26:59.721608 21471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:26:59.721774 21456 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:26:59.721856 21456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "53e5f6ac74314dbc880a0192d1c511c1"
format_stamp: "Formatted at 2026-09-27 03:26:59 on dist-test-slave-gw2v"
server_key: "3b45e4df364b58184057c81fabe06502"
server_key_iv: "20f2d56ac997d8abb9cebd2e88299a2f"
server_key_version: "encryptionkey@0"
I20260927 03:26:59.721915 21456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:26:59.725383 21456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:26:59.727511 21456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:26:59.727648 21456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:26:59.731547 21456 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40187
I20260927 03:26:59.731587 21523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40187 every 8 connection(s)
I20260927 03:26:59.731889 21456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:26:59.732828 21524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:26:59.735167 21524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1: Bootstrap starting.
I20260927 03:26:59.735761 21524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1: Neither blocks nor log segments found. Creating new log.
I20260927 03:26:59.735838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 21456
I20260927 03:26:59.735972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:26:59.736073 21524 log.cc:824] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1: Log is configured to *not* fsync() on all Append() calls
I20260927 03:26:59.736150 392 external_mini_cluster.cc:1428] Setting key 116fcef51c6172326a7de23581ca4f28
I20260927 03:26:59.736958 21524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1: No bootstrap required, opened a new log
I20260927 03:26:59.738619 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:59.736690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:36193 (local address 127.0.0.1:40187)
0927 03:26:59.736863 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:59.736873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:59.736899 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:26:59.737424 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:59.737433 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:59.737467 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:59.737586 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:59.738195 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:59.738198 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:59.738198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:59.738202 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:59.738241 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:59.738252 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:59.738319 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:59.738384 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:26:59.738420 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260927 03:26:59.739547 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:26:59.739562 21524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e5f6ac74314dbc880a0192d1c511c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40187 } }
I20260927 03:26:59.739722 21524 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:26:59.739759 21524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e5f6ac74314dbc880a0192d1c511c1, State: Initialized, Role: FOLLOWER
I20260927 03:26:59.739878 21524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e5f6ac74314dbc880a0192d1c511c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40187 } }
I20260927 03:26:59.739950 21524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:26:59.739972 21524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:26:59.740016 21524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:26:59.740594 21524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e5f6ac74314dbc880a0192d1c511c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40187 } }
I20260927 03:26:59.740715 21524 leader_election.cc:302] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53e5f6ac74314dbc880a0192d1c511c1; no voters:
I20260927 03:26:59.740882 21524 leader_election.cc:288] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:26:59.740931 21533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:26:59.741125 21533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [term 1 LEADER]: Becoming Leader. State: Replica: 53e5f6ac74314dbc880a0192d1c511c1, State: Running, Role: LEADER
I20260927 03:26:59.741182 21524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:26:59.741230 21533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e5f6ac74314dbc880a0192d1c511c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40187 } }
I20260927 03:26:59.741555 21533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53e5f6ac74314dbc880a0192d1c511c1. Latest consensus state: current_term: 1 leader_uuid: "53e5f6ac74314dbc880a0192d1c511c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e5f6ac74314dbc880a0192d1c511c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40187 } } }
I20260927 03:26:59.741665 21533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:59.741643 21534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53e5f6ac74314dbc880a0192d1c511c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e5f6ac74314dbc880a0192d1c511c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40187 } } }
I20260927 03:26:59.741765 21534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1 [sys.catalog]: This master's current role is: LEADER
I20260927 03:26:59.742190 21542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:26:59.742777 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:26:59.740954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:55593 (local address 127.0.0.1:40187)
0927 03:26:59.740995 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:26:59.740997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:26:59.741699 (+ 702us) server_negotiation.cc:407] Connection header received
0927 03:26:59.741828 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:26:59.741829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:26:59.741834 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:26:59.741862 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:26:59.742551 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:26:59.742552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:26:59.742552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:26:59.742556 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:26:59.742572 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:26:59.742576 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:26:59.742631 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:26:59.742694 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 03:26:59.742720 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:26:59.742926 21542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:26:59.744647 21542 catalog_manager.cc:1347] Generated new cluster ID: 0a25c70b23a74073b5620541ad92daff
I20260927 03:26:59.744697 21542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:26:59.752573 21542 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:26:59.752738 21542 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:26:59.759789 21542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53e5f6ac74314dbc880a0192d1c511c1: Generated new TSK 0
I20260927 03:26:59.759999 21542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:26:59.763854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 21456
2026-09-27T03:26:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:26:59 dist-test-slave-gw2v krb5kdc[21555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:26:59 dist-test-slave-gw2v krb5kdc[21555](info): set up 2 sockets
Sep 27 03:26:59 dist-test-slave-gw2v krb5kdc[21555](info): commencing operation
krb5kdc: starting...
W20260927 03:27:00.718869 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:27:00 dist-test-slave-gw2v krb5kdc[21555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:27:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:27:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:27:00.838330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:27:00.919740 21568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:00.919967 21568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:00.919999 21568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:00.921178 21568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:00.921242 21568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:00.921263 21568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:00.921279 21568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:00.921303 21568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:00.922894 21568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:00.923177 21568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:00.923405 21568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:00.928157 21573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:00.928146 21576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:00.928146 21574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:00.928350 21568 server_base.cc:1034] running on GCE node
I20260927 03:27:00.928635 21568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:00.928905 21568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:00.930073 21568 hybrid_clock.cc:648] HybridClock initialized: now 1790479620930054 us; error 30 us; skew 500 ppm
Sep 27 03:27:00 dist-test-slave-gw2v krb5kdc[21555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:00.933189 21568 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:27:00.933660 21568 webserver.cc:460] Webserver started at http://127.0.0.1:46241/ using document root <none> and password file <none>
I20260927 03:27:00.933880 21568 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:00.933928 21568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:00.934053 21568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:00.935312 21568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "17a3e06ac78a49569ee786ec59157aef"
format_stamp: "Formatted at 2026-09-27 03:27:00 on dist-test-slave-gw2v"
server_key: "a9598a37d140e8e6856044f15209c02a"
server_key_iv: "528bb5b5799dabda92492b06fef00f01"
server_key_version: "encryptionkey@0"
I20260927 03:27:00.935672 21568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "17a3e06ac78a49569ee786ec59157aef"
format_stamp: "Formatted at 2026-09-27 03:27:00 on dist-test-slave-gw2v"
server_key: "a9598a37d140e8e6856044f15209c02a"
server_key_iv: "528bb5b5799dabda92492b06fef00f01"
server_key_version: "encryptionkey@0"
I20260927 03:27:00.936950 21568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:27:00.937692 21583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:00.937844 21568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:27:00.937927 21568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "17a3e06ac78a49569ee786ec59157aef"
format_stamp: "Formatted at 2026-09-27 03:27:00 on dist-test-slave-gw2v"
server_key: "a9598a37d140e8e6856044f15209c02a"
server_key_iv: "528bb5b5799dabda92492b06fef00f01"
server_key_version: "encryptionkey@0"
I20260927 03:27:00.937992 21568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:00.949580 21568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:00.951427 21568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:00.951561 21568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:00.955339 21568 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45947
I20260927 03:27:00.955395 21635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45947 every 8 connection(s)
I20260927 03:27:00.955701 21568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:27:00.956490 21636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:00.958721 21636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef: Bootstrap starting.
I20260927 03:27:00.959301 21636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:00.959551 21636 log.cc:824] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:00.960201 21636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef: No bootstrap required, opened a new log
I20260927 03:27:00.961802 21636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a3e06ac78a49569ee786ec59157aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } }
I20260927 03:27:00.961942 21636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:27:00.961963 21636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a3e06ac78a49569ee786ec59157aef, State: Initialized, Role: FOLLOWER
I20260927 03:27:00.962088 21636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a3e06ac78a49569ee786ec59157aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } }
I20260927 03:27:00.962152 21636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:27:00.962172 21636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:27:00.962198 21636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:00.962720 21636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a3e06ac78a49569ee786ec59157aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } }
I20260927 03:27:00.962874 21636 leader_election.cc:302] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17a3e06ac78a49569ee786ec59157aef; no voters:
I20260927 03:27:00.962927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 21568
I20260927 03:27:00.963037 21636 leader_election.cc:288] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:27:00.963055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:27:00.963091 21639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:27:00.963246 21639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [term 1 LEADER]: Becoming Leader. State: Replica: 17a3e06ac78a49569ee786ec59157aef, State: Running, Role: LEADER
I20260927 03:27:00.963254 392 external_mini_cluster.cc:1428] Setting key 8373a01dfb6ac2ccaf4a6edb7823ea00
I20260927 03:27:00.963286 21636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:27:00.963392 21639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a3e06ac78a49569ee786ec59157aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } }
I20260927 03:27:00.963898 21641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17a3e06ac78a49569ee786ec59157aef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a3e06ac78a49569ee786ec59157aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } } }
I20260927 03:27:00.964046 21640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17a3e06ac78a49569ee786ec59157aef. Latest consensus state: current_term: 1 leader_uuid: "17a3e06ac78a49569ee786ec59157aef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a3e06ac78a49569ee786ec59157aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } } }
I20260927 03:27:00.964102 21641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:00.964186 21640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:00.964365 21650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:27:00.964852 21650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:27:00.968168 21650 catalog_manager.cc:1347] Generated new cluster ID: f42b0f9dca6048d5a0e0f2012c150357
I20260927 03:27:00.968235 21650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:27:00 dist-test-slave-gw2v krb5kdc[21555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 03:27:00.974936 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:00.963806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:50921 (local address 127.0.0.1:45947)
0927 03:27:00.965629 (+ 1823us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:00.965638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:00.965671 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:27:00.965784 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:00.965791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:00.965849 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:00.965996 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:00.966622 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:00.968027 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:00.968895 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:00.969339 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:00.969439 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:00.971786 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:00.971791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:00.971791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:00.971805 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:00.973369 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:00.973744 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:00.973747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:00.973747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:00.973846 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:00.974054 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:00.974055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:00.974055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:00.974300 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:00.974482 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:00.974568 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 03:27:00.974669 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1774,"thread_start_us":381,"threads_started":1}
I20260927 03:27:00.975870 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:27:00.986780 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:00.977164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:39539 (local address 127.0.0.1:45947)
0927 03:27:00.977226 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:00.977228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:00.981127 (+ 3899us) server_negotiation.cc:407] Connection header received
0927 03:27:00.981188 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:00.981189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:00.981200 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:00.981257 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:00.981769 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:00.982325 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:00.983082 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:00.983297 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:00.983351 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:00.984477 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:00.984478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:00.984479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:00.984486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:00.985713 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:00.986040 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:00.986041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:00.986042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:00.986090 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:00.986301 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:00.986302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:00.986302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:00.986402 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:00.986486 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:00.986623 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 03:27:00.986683 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 03:27:01.004846 21650 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:27:01.005375 21650 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:27:01.011104 21650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17a3e06ac78a49569ee786ec59157aef: Generated new TSK 0
I20260927 03:27:01.011317 21650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:27:01.017149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 21568
W20260927 03:27:01.021167 21553 connection.cc:535] client connection to 127.0.0.1:45947 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45947): Connection reset by peer (error 104)
2026-09-27T03:27:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:27:01 dist-test-slave-gw2v krb5kdc[21667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:27:01 dist-test-slave-gw2v krb5kdc[21667](info): set up 2 sockets
Sep 27 03:27:01 dist-test-slave-gw2v krb5kdc[21667](info): commencing operation
krb5kdc: starting...
W20260927 03:27:01.978778 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:27:02 dist-test-slave-gw2v krb5kdc[21667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:27:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:27:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:27:02.102535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34389
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:27:02.182600 21680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:02.182859 21680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:02.182884 21680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:02.184032 21680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:02.184072 21680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:02.184084 21680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:02.184098 21680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:02.184108 21680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:02.185561 21680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34389
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:02.185770 21680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:02.185964 21680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:02.190877 21688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:02.190898 21686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:02.190871 21685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:02.191090 21680 server_base.cc:1034] running on GCE node
I20260927 03:27:02.191452 21680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:02.191745 21680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:02.192914 21680 hybrid_clock.cc:648] HybridClock initialized: now 1790479622192883 us; error 46 us; skew 500 ppm
Sep 27 03:27:02 dist-test-slave-gw2v krb5kdc[21667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:02.195603 21680 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:27:02.196085 21680 webserver.cc:460] Webserver started at http://127.0.0.1:35979/ using document root <none> and password file <none>
I20260927 03:27:02.196399 21680 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:02.196455 21680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:02.196583 21680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:02.197676 21680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "cfe7456770954a5b93d3f7f65e90a121"
format_stamp: "Formatted at 2026-09-27 03:27:02 on dist-test-slave-gw2v"
server_key: "574ee398c8bfcc8c2a4a33cc3c99917e"
server_key_iv: "49492b438039785149c65a12fa61cd9f"
server_key_version: "encryptionkey@0"
I20260927 03:27:02.197997 21680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "cfe7456770954a5b93d3f7f65e90a121"
format_stamp: "Formatted at 2026-09-27 03:27:02 on dist-test-slave-gw2v"
server_key: "574ee398c8bfcc8c2a4a33cc3c99917e"
server_key_iv: "49492b438039785149c65a12fa61cd9f"
server_key_version: "encryptionkey@0"
I20260927 03:27:02.199424 21680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:27:02.200206 21695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:02.200398 21680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:27:02.200487 21680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "cfe7456770954a5b93d3f7f65e90a121"
format_stamp: "Formatted at 2026-09-27 03:27:02 on dist-test-slave-gw2v"
server_key: "574ee398c8bfcc8c2a4a33cc3c99917e"
server_key_iv: "49492b438039785149c65a12fa61cd9f"
server_key_version: "encryptionkey@0"
I20260927 03:27:02.200552 21680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:02.213848 21680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:02.216300 21680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:02.216459 21680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:02.220472 21680 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34389
I20260927 03:27:02.220553 21747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34389 every 8 connection(s)
I20260927 03:27:02.220873 21680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:27:02.221894 21748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:02.224254 21748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121: Bootstrap starting.
I20260927 03:27:02.224841 21748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:02.225095 21748 log.cc:824] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:02.225859 21748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121: No bootstrap required, opened a new log
I20260927 03:27:02.227499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 21680
I20260927 03:27:02.227605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:27:02.227623 21748 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe7456770954a5b93d3f7f65e90a121" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34389 } }
I20260927 03:27:02.227806 392 external_mini_cluster.cc:1428] Setting key 7d64c9b2e295e6a6006019e616b3bb54
I20260927 03:27:02.227795 21748 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:27:02.227835 21748 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfe7456770954a5b93d3f7f65e90a121, State: Initialized, Role: FOLLOWER
I20260927 03:27:02.227993 21748 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe7456770954a5b93d3f7f65e90a121" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34389 } }
I20260927 03:27:02.228076 21748 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:27:02.228104 21748 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:27:02.228152 21748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:02.229225 21748 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe7456770954a5b93d3f7f65e90a121" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34389 } }
I20260927 03:27:02.229323 21748 leader_election.cc:302] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfe7456770954a5b93d3f7f65e90a121; no voters:
I20260927 03:27:02.229480 21748 leader_election.cc:288] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:27:02.229528 21753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:27:02.229667 21753 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [term 1 LEADER]: Becoming Leader. State: Replica: cfe7456770954a5b93d3f7f65e90a121, State: Running, Role: LEADER
I20260927 03:27:02.229740 21748 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:27:02.229794 21753 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe7456770954a5b93d3f7f65e90a121" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34389 } }
I20260927 03:27:02.230136 21753 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfe7456770954a5b93d3f7f65e90a121. Latest consensus state: current_term: 1 leader_uuid: "cfe7456770954a5b93d3f7f65e90a121" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe7456770954a5b93d3f7f65e90a121" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34389 } } }
I20260927 03:27:02.230356 21753 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:02.230284 21755 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfe7456770954a5b93d3f7f65e90a121" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe7456770954a5b93d3f7f65e90a121" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34389 } } }
I20260927 03:27:02.230588 21755 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121 [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:02.231032 21762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:27:02.231961 21762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:27:02.233858 21762 catalog_manager.cc:1347] Generated new cluster ID: b68c1c89d9ab43a5ac27e7e229472905
I20260927 03:27:02.233907 21762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:27:02.234169 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:02.228376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:37941 (local address 127.0.0.1:34389)
0927 03:27:02.228591 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:02.228602 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:02.228641 (+ 39us) server_negotiation.cc:407] Connection header received
0927 03:27:02.228784 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:02.228792 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:02.228858 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:02.229011 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:02.229380 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:02.230795 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:02.231812 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:02.232570 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:02.232685 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:02.233416 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:02.233420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:02.233420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:27:02.233626 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:02.233674 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:27:02.233690 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:02.233771 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:02.233919 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 03:27:02.233948 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":108,"threads_started":1}
I20260927 03:27:02.235165 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:27:02.240080 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:02.236603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:56519 (local address 127.0.0.1:34389)
0927 03:27:02.236640 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:02.236641 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:02.236711 (+ 70us) server_negotiation.cc:407] Connection header received
0927 03:27:02.236755 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:02.236756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:02.236765 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:02.236834 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:02.236958 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:02.237739 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:02.238716 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:02.239067 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:02.239139 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:02.239831 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:02.239833 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:02.239833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:27:02.239839 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:02.239855 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:27:02.239861 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:02.239925 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:02.239981 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 03:27:02.239995 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:27:02.251825 21762 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:27:02.252532 21762 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:27:02.256132 21762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfe7456770954a5b93d3f7f65e90a121: Generated new TSK 0
I20260927 03:27:02.256357 21762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:27:02.259410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 21680
W20260927 03:27:02.263252 21665 connection.cc:535] client connection to 127.0.0.1:34389 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34389): Connection reset by peer (error 104)
2026-09-27T03:27:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1242 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:27:02 dist-test-slave-gw2v krb5kdc[21779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:27:02 dist-test-slave-gw2v krb5kdc[21779](info): set up 2 sockets
Sep 27 03:27:02 dist-test-slave-gw2v krb5kdc[21779](info): commencing operation
krb5kdc: starting...
W20260927 03:27:03.202755 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:27:03 dist-test-slave-gw2v krb5kdc[21779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:27:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:27:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:27:03.315827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:27:03.396677 21792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:03.396890 21792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:03.396929 21792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:03.398080 21792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:03.398123 21792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:03.398141 21792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:03.398157 21792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:03.398172 21792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:03.399646 21792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40837
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:03.399883 21792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:03.400111 21792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:03.404814 21797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:03.404783 21798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:03.404772 21800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:03.405155 21792 server_base.cc:1034] running on GCE node
I20260927 03:27:03.405375 21792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:03.405633 21792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:03.406769 21792 hybrid_clock.cc:648] HybridClock initialized: now 1790479623406753 us; error 32 us; skew 500 ppm
Sep 27 03:27:03 dist-test-slave-gw2v krb5kdc[21779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:03.409142 21792 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:27:03.409533 21792 webserver.cc:460] Webserver started at http://127.0.0.1:43231/ using document root <none> and password file <none>
I20260927 03:27:03.409722 21792 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:03.409760 21792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:03.409849 21792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:03.410956 21792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "8c9c60da5e5a4859907b9872a5614e7f"
format_stamp: "Formatted at 2026-09-27 03:27:03 on dist-test-slave-gw2v"
server_key: "63bca374d9ba03797f6fd1be89e8400d"
server_key_iv: "b06e010372f34ad10b08ed32016ef8cf"
server_key_version: "encryptionkey@0"
I20260927 03:27:03.411262 21792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c9c60da5e5a4859907b9872a5614e7f"
format_stamp: "Formatted at 2026-09-27 03:27:03 on dist-test-slave-gw2v"
server_key: "63bca374d9ba03797f6fd1be89e8400d"
server_key_iv: "b06e010372f34ad10b08ed32016ef8cf"
server_key_version: "encryptionkey@0"
I20260927 03:27:03.412515 21792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:27:03.413188 21807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:03.413354 21792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:27:03.413436 21792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "8c9c60da5e5a4859907b9872a5614e7f"
format_stamp: "Formatted at 2026-09-27 03:27:03 on dist-test-slave-gw2v"
server_key: "63bca374d9ba03797f6fd1be89e8400d"
server_key_iv: "b06e010372f34ad10b08ed32016ef8cf"
server_key_version: "encryptionkey@0"
I20260927 03:27:03.413493 21792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:03.416723 21792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:03.418780 21792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:03.418926 21792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:03.422446 21792 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40837
I20260927 03:27:03.422508 21859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40837 every 8 connection(s)
I20260927 03:27:03.422777 21792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:27:03.423725 21860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:03.425923 21860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f: Bootstrap starting.
I20260927 03:27:03.426477 21860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:03.426729 21860 log.cc:824] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:03.427399 21860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f: No bootstrap required, opened a new log
I20260927 03:27:03.428996 21860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9c60da5e5a4859907b9872a5614e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260927 03:27:03.429136 21860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:27:03.429167 21860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9c60da5e5a4859907b9872a5614e7f, State: Initialized, Role: FOLLOWER
I20260927 03:27:03.429307 21860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9c60da5e5a4859907b9872a5614e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260927 03:27:03.429392 21860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:27:03.429422 21860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:27:03.429468 21860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:03.429996 21860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9c60da5e5a4859907b9872a5614e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260927 03:27:03.430081 21860 leader_election.cc:302] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c9c60da5e5a4859907b9872a5614e7f; no voters:
I20260927 03:27:03.430246 21860 leader_election.cc:288] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:27:03.430334 21863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:27:03.430356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 21792
I20260927 03:27:03.430459 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:27:03.430531 21863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [term 1 LEADER]: Becoming Leader. State: Replica: 8c9c60da5e5a4859907b9872a5614e7f, State: Running, Role: LEADER
I20260927 03:27:03.430645 392 external_mini_cluster.cc:1428] Setting key 4996895ef39029535545fb94a3c26a27
I20260927 03:27:03.430536 21860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:27:03.430686 21863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9c60da5e5a4859907b9872a5614e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260927 03:27:03.431093 21864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c9c60da5e5a4859907b9872a5614e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9c60da5e5a4859907b9872a5614e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } } }
I20260927 03:27:03.431167 21864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:03.431365 21865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c9c60da5e5a4859907b9872a5614e7f. Latest consensus state: current_term: 1 leader_uuid: "8c9c60da5e5a4859907b9872a5614e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9c60da5e5a4859907b9872a5614e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } } }
I20260927 03:27:03.431711 21865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:03.431818 21868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:27:03.432428 21868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:27:03.433324 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:03.431333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:43327 (local address 127.0.0.1:40837)
0927 03:27:03.431573 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:03.431585 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:03.431772 (+ 187us) server_negotiation.cc:407] Connection header received
0927 03:27:03.431944 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:03.431953 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:03.431986 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:03.432134 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:03.432882 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:03.432886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:03.432886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:27:03.432890 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:03.432934 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:27:03.432948 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:03.433004 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:03.433081 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 03:27:03.433139 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":64,"threads_started":1}
I20260927 03:27:03.434192 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:27:03.434371 21868 catalog_manager.cc:1347] Generated new cluster ID: c6fd477d1e094d03b11c5d0f68d15333
I20260927 03:27:03.434429 21868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:27:03.442226 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:03.436599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:38497 (local address 127.0.0.1:40837)
0927 03:27:03.441129 (+ 4530us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:03.441131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:03.441144 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:27:03.441171 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:03.441172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:03.441177 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:03.441257 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:03.441961 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:03.441962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:03.441962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:27:03.441968 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:03.441986 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:27:03.441991 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:03.442041 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:03.442112 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 03:27:03.442127 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4502}
I20260927 03:27:03.448549 21868 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:27:03.448745 21868 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:27:03.455317 21868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c9c60da5e5a4859907b9872a5614e7f: Generated new TSK 0
I20260927 03:27:03.455536 21868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:27:03.461139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 21792
2026-09-27T03:27:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:27:03 dist-test-slave-gw2v krb5kdc[21891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:27:03 dist-test-slave-gw2v krb5kdc[21891](info): set up 2 sockets
Sep 27 03:27:03 dist-test-slave-gw2v krb5kdc[21891](info): commencing operation
krb5kdc: starting...
W20260927 03:27:04.414669 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:27:04 dist-test-slave-gw2v krb5kdc[21891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:27:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:27:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:27:04.525380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42815
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:27:04.605530 21904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:04.605746 21904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:04.605767 21904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:04.607003 21904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:04.607049 21904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:04.607064 21904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:04.607074 21904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:04.607084 21904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:04.608572 21904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:04.608810 21904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:04.609021 21904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:04.613780 21912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:04.613860 21910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:04.613802 21909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:04.614059 21904 server_base.cc:1034] running on GCE node
I20260927 03:27:04.614313 21904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:04.614723 21904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:04.615900 21904 hybrid_clock.cc:648] HybridClock initialized: now 1790479624615883 us; error 28 us; skew 500 ppm
Sep 27 03:27:04 dist-test-slave-gw2v krb5kdc[21891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:04.618510 21904 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:27:04.618986 21904 webserver.cc:460] Webserver started at http://127.0.98.62:34817/ using document root <none> and password file <none>
I20260927 03:27:04.619197 21904 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:04.619242 21904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:04.619350 21904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:04.620393 21904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "941ac8d6778645e386db437b681f9f72"
format_stamp: "Formatted at 2026-09-27 03:27:04 on dist-test-slave-gw2v"
server_key: "ecc1cddf54b364ed928500ffdf6278c5"
server_key_iv: "d2aa5faaa6c8b4bde13c7ae83f32b630"
server_key_version: "encryptionkey@0"
I20260927 03:27:04.620714 21904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "941ac8d6778645e386db437b681f9f72"
format_stamp: "Formatted at 2026-09-27 03:27:04 on dist-test-slave-gw2v"
server_key: "ecc1cddf54b364ed928500ffdf6278c5"
server_key_iv: "d2aa5faaa6c8b4bde13c7ae83f32b630"
server_key_version: "encryptionkey@0"
I20260927 03:27:04.621953 21904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:27:04.622725 21919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:04.622889 21904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:27:04.622970 21904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "941ac8d6778645e386db437b681f9f72"
format_stamp: "Formatted at 2026-09-27 03:27:04 on dist-test-slave-gw2v"
server_key: "ecc1cddf54b364ed928500ffdf6278c5"
server_key_iv: "d2aa5faaa6c8b4bde13c7ae83f32b630"
server_key_version: "encryptionkey@0"
I20260927 03:27:04.623039 21904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:04.645236 21904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:04.647398 21904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:04.647513 21904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:04.651329 21904 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42815
I20260927 03:27:04.651383 21971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42815 every 8 connection(s)
I20260927 03:27:04.651714 21904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:27:04.652613 21972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:04.655040 21972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72: Bootstrap starting.
I20260927 03:27:04.655624 21972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:04.655902 21972 log.cc:824] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:04.656746 21972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72: No bootstrap required, opened a new log
I20260927 03:27:04.658481 21972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941ac8d6778645e386db437b681f9f72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } }
I20260927 03:27:04.658643 21972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:27:04.658675 21972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941ac8d6778645e386db437b681f9f72, State: Initialized, Role: FOLLOWER
I20260927 03:27:04.658809 21972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941ac8d6778645e386db437b681f9f72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } }
I20260927 03:27:04.658915 21972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:27:04.658947 21972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:27:04.658998 21972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:04.659591 21972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941ac8d6778645e386db437b681f9f72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } }
I20260927 03:27:04.659698 21972 leader_election.cc:302] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 941ac8d6778645e386db437b681f9f72; no voters:
I20260927 03:27:04.659879 21972 leader_election.cc:288] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:27:04.659950 21975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:27:04.660157 21972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:27:04.660153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 21904
I20260927 03:27:04.660260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:27:04.660180 21975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [term 1 LEADER]: Becoming Leader. State: Replica: 941ac8d6778645e386db437b681f9f72, State: Running, Role: LEADER
I20260927 03:27:04.660455 392 external_mini_cluster.cc:1428] Setting key c6ebe7f57e994ec7b8af2ad5f54852ef
I20260927 03:27:04.660461 21975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941ac8d6778645e386db437b681f9f72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } }
I20260927 03:27:04.661007 21977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 941ac8d6778645e386db437b681f9f72. Latest consensus state: current_term: 1 leader_uuid: "941ac8d6778645e386db437b681f9f72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941ac8d6778645e386db437b681f9f72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } } }
I20260927 03:27:04.661088 21977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:04.661417 21976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "941ac8d6778645e386db437b681f9f72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941ac8d6778645e386db437b681f9f72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } } }
I20260927 03:27:04.661504 21976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72 [sys.catalog]: This master's current role is: LEADER
W20260927 03:27:04.662086 21993 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:27:04.662142 21993 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:27:04.662189 21986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:27:04.662343 21986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:27:04.663833 21986 catalog_manager.cc:1347] Generated new cluster ID: 8cb92b2242d1408a80c918fe759b1e42
I20260927 03:27:04.663882 21986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:27:04 dist-test-slave-gw2v krb5kdc[21891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:27:04.672721 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:04.661104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49103 (local address 127.0.98.62:42815)
0927 03:27:04.661290 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:04.661302 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:04.661973 (+ 671us) server_negotiation.cc:407] Connection header received
0927 03:27:04.662105 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:04.662111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:04.662173 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:04.662313 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:04.662960 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:04.664095 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:04.665178 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:04.665792 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:04.665885 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:04.668744 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:04.668750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:04.668750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:04.668767 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:04.670795 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:04.671276 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:04.671280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:04.671280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:04.671416 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:04.671635 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:04.671636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:04.671636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:04.671988 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:04.672167 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:04.672249 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 03:27:04.672477 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260927 03:27:04.673775 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:27:04.675582 21986 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:27:04.676085 21986 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:27:04.678807 21986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 941ac8d6778645e386db437b681f9f72: Generated new TSK 0
I20260927 03:27:04.679039 21986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:27:04.682098 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:04.675434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39305 (local address 127.0.98.62:42815)
0927 03:27:04.675500 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:04.675502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:04.675511 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:27:04.675531 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:04.675532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:04.675540 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:04.675597 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:04.676314 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:04.677023 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:04.678029 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:04.678355 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:04.678417 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:04.679868 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:04.679869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:04.679870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:04.679876 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:04.681082 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:04.681460 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:04.681461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:04.681461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:04.681505 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:04.681702 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:04.681702 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:04.681702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:04.681790 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:04.681859 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:04.681992 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 03:27:04.682035 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":50}
I20260927 03:27:04.684113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 21904
W20260927 03:27:04.687883 21889 connection.cc:535] client connection to 127.0.98.62:42815 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42815): Connection reset by peer (error 104)
2026-09-27T03:27:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:27:04 dist-test-slave-gw2v krb5kdc[22003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:27:04 dist-test-slave-gw2v krb5kdc[22003](info): set up 2 sockets
Sep 27 03:27:04 dist-test-slave-gw2v krb5kdc[22003](info): commencing operation
krb5kdc: starting...
W20260927 03:27:05.638803 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:27:05 dist-test-slave-gw2v krb5kdc[22003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:27:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:27:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:27:05.762301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45251
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:27:05.843101 22016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:05.843327 22016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:05.843357 22016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:05.844548 22016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:05.844592 22016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:05.844610 22016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:05.844625 22016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:05.844640 22016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:05.846127 22016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45251
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:05.846376 22016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:05.846577 22016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:05.851274 22024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:05.851284 22021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:05.851284 22022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:05.851613 22016 server_base.cc:1034] running on GCE node
I20260927 03:27:05.851843 22016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:05.852119 22016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:05.853284 22016 hybrid_clock.cc:648] HybridClock initialized: now 1790479625853269 us; error 30 us; skew 500 ppm
Sep 27 03:27:05 dist-test-slave-gw2v krb5kdc[22003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:05.855791 22016 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:27:05.856257 22016 webserver.cc:460] Webserver started at http://127.0.98.62:44665/ using document root <none> and password file <none>
I20260927 03:27:05.856467 22016 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:05.856516 22016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:05.856643 22016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:05.857687 22016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "cc8dac67333542229fb402a2a428d7cb"
format_stamp: "Formatted at 2026-09-27 03:27:05 on dist-test-slave-gw2v"
server_key: "cf382ba3c0ef294a78c765a23063a949"
server_key_iv: "f16c73bb56c99f5383cfdd773be6b887"
server_key_version: "encryptionkey@0"
I20260927 03:27:05.858009 22016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc8dac67333542229fb402a2a428d7cb"
format_stamp: "Formatted at 2026-09-27 03:27:05 on dist-test-slave-gw2v"
server_key: "cf382ba3c0ef294a78c765a23063a949"
server_key_iv: "f16c73bb56c99f5383cfdd773be6b887"
server_key_version: "encryptionkey@0"
I20260927 03:27:05.859328 22016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:27:05.860083 22031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:05.860249 22016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:27:05.860332 22016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "cc8dac67333542229fb402a2a428d7cb"
format_stamp: "Formatted at 2026-09-27 03:27:05 on dist-test-slave-gw2v"
server_key: "cf382ba3c0ef294a78c765a23063a949"
server_key_iv: "f16c73bb56c99f5383cfdd773be6b887"
server_key_version: "encryptionkey@0"
I20260927 03:27:05.860391 22016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:05.872162 22016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:05.874300 22016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:05.874444 22016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:05.878525 22016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45251
I20260927 03:27:05.878662 22083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45251 every 8 connection(s)
I20260927 03:27:05.878961 22016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:27:05.879823 22084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:05.882189 22084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb: Bootstrap starting.
I20260927 03:27:05.882751 22084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:05.883034 22084 log.cc:824] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:05.883736 22084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb: No bootstrap required, opened a new log
I20260927 03:27:05.885351 22084 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8dac67333542229fb402a2a428d7cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45251 } }
I20260927 03:27:05.885499 22084 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:27:05.885521 22084 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8dac67333542229fb402a2a428d7cb, State: Initialized, Role: FOLLOWER
I20260927 03:27:05.885643 22084 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8dac67333542229fb402a2a428d7cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45251 } }
I20260927 03:27:05.885715 22084 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:27:05.885737 22084 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:27:05.885783 22084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:05.886315 22084 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8dac67333542229fb402a2a428d7cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45251 } }
I20260927 03:27:05.886404 22084 leader_election.cc:302] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc8dac67333542229fb402a2a428d7cb; no voters:
I20260927 03:27:05.886559 22084 leader_election.cc:288] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:27:05.886628 22087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:27:05.886790 22087 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [term 1 LEADER]: Becoming Leader. State: Replica: cc8dac67333542229fb402a2a428d7cb, State: Running, Role: LEADER
I20260927 03:27:05.886816 22084 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:27:05.886914 22087 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8dac67333542229fb402a2a428d7cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45251 } }
I20260927 03:27:05.887387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 22016
I20260927 03:27:05.887501 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:27:05.887488 22088 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc8dac67333542229fb402a2a428d7cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8dac67333542229fb402a2a428d7cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45251 } } }
I20260927 03:27:05.887560 22088 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:05.887712 392 external_mini_cluster.cc:1428] Setting key e5120189eac5036052ed4f881a498363
I20260927 03:27:05.887712 22089 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc8dac67333542229fb402a2a428d7cb. Latest consensus state: current_term: 1 leader_uuid: "cc8dac67333542229fb402a2a428d7cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8dac67333542229fb402a2a428d7cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45251 } } }
I20260927 03:27:05.887773 22089 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:05.887953 22096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:27:05.888783 22096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:27:05.890326 22096 catalog_manager.cc:1347] Generated new cluster ID: a3af3a0112f84dddaff29a555e821956
I20260927 03:27:05.890377 22096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:27:05.893903 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:05.888242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52159 (local address 127.0.98.62:45251)
0927 03:27:05.888665 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:05.888676 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:05.888713 (+ 37us) server_negotiation.cc:407] Connection header received
0927 03:27:05.888838 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:05.888845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:05.888904 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:05.889056 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:05.889437 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:05.890658 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:05.891788 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:05.892397 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:05.892492 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:05.893226 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:05.893231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:05.893231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:27:05.893431 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:05.893468 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:27:05.893482 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:05.893560 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:05.893655 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 03:27:05.893681 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":66,"threads_started":1}
I20260927 03:27:05.894868 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:27:05.901748 22096 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:27:05.902431 22096 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:27:05.902683 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:05.896292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59843 (local address 127.0.98.62:45251)
0927 03:27:05.896323 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:05.896325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:05.899102 (+ 2777us) server_negotiation.cc:407] Connection header received
0927 03:27:05.899174 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:05.899175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:05.899204 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:05.899288 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:05.899423 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:05.900235 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:05.901392 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:05.901630 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:05.901691 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:05.902420 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:05.902421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:05.902422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:27:05.902426 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:05.902441 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:27:05.902446 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:05.902476 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:05.902595 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 03:27:05.902608 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 03:27:05.910291 22096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc8dac67333542229fb402a2a428d7cb: Generated new TSK 0
I20260927 03:27:05.910493 22096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:27:05.919021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 22016
W20260927 03:27:05.922778 22001 connection.cc:535] client connection to 127.0.98.62:45251 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:45251): Connection reset by peer (error 104)
2026-09-27T03:27:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1234 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:27:05 dist-test-slave-gw2v krb5kdc[22115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:27:05 dist-test-slave-gw2v krb5kdc[22115](info): set up 2 sockets
Sep 27 03:27:05 dist-test-slave-gw2v krb5kdc[22115](info): commencing operation
krb5kdc: starting...
W20260927 03:27:06.870666 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:27:06 dist-test-slave-gw2v krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:27:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:27:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:27:06.986892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41577
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:27:07.066694 22128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:07.066946 22128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:07.066980 22128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:07.068179 22128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:07.068223 22128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:07.068243 22128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:07.068259 22128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:07.068274 22128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:07.069751 22128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41577
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:07.070003 22128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:07.070228 22128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:07.075125 22133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:07.075137 22136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:07.075191 22128 server_base.cc:1034] running on GCE node
W20260927 03:27:07.075137 22134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:07.075558 22128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:07.075832 22128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:07.077018 22128 hybrid_clock.cc:648] HybridClock initialized: now 1790479627076970 us; error 50 us; skew 500 ppm
Sep 27 03:27:07 dist-test-slave-gw2v krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:07.079573 22128 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:27:07.080027 22128 webserver.cc:460] Webserver started at http://127.0.98.62:44389/ using document root <none> and password file <none>
I20260927 03:27:07.080250 22128 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:07.080302 22128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:07.080430 22128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:07.081573 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "5af9d316295641a78bca907d964a26bd"
format_stamp: "Formatted at 2026-09-27 03:27:07 on dist-test-slave-gw2v"
server_key: "e3faf16e7ab994fdb5df907d71231c87"
server_key_iv: "20edd29a8b4ddaaf7dbea7bb42a239c3"
server_key_version: "encryptionkey@0"
I20260927 03:27:07.081924 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "5af9d316295641a78bca907d964a26bd"
format_stamp: "Formatted at 2026-09-27 03:27:07 on dist-test-slave-gw2v"
server_key: "e3faf16e7ab994fdb5df907d71231c87"
server_key_iv: "20edd29a8b4ddaaf7dbea7bb42a239c3"
server_key_version: "encryptionkey@0"
I20260927 03:27:07.083299 22128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:27:07.084141 22143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:07.084352 22128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:27:07.084446 22128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "5af9d316295641a78bca907d964a26bd"
format_stamp: "Formatted at 2026-09-27 03:27:07 on dist-test-slave-gw2v"
server_key: "e3faf16e7ab994fdb5df907d71231c87"
server_key_iv: "20edd29a8b4ddaaf7dbea7bb42a239c3"
server_key_version: "encryptionkey@0"
I20260927 03:27:07.084510 22128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:07.087949 22128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:07.089753 22128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:07.089882 22128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:07.093492 22128 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41577
I20260927 03:27:07.093533 22195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41577 every 8 connection(s)
I20260927 03:27:07.093834 22128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:27:07.094751 22196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:07.097038 22196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd: Bootstrap starting.
I20260927 03:27:07.097755 22196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:07.097998 22196 log.cc:824] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:07.098663 22196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd: No bootstrap required, opened a new log
I20260927 03:27:07.100278 22196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af9d316295641a78bca907d964a26bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41577 } }
I20260927 03:27:07.100417 22196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:27:07.100438 22196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af9d316295641a78bca907d964a26bd, State: Initialized, Role: FOLLOWER
I20260927 03:27:07.100549 22196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af9d316295641a78bca907d964a26bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41577 } }
I20260927 03:27:07.100628 22196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:27:07.100652 22196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:27:07.100685 22196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:07.101171 22196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af9d316295641a78bca907d964a26bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41577 } }
I20260927 03:27:07.101256 22196 leader_election.cc:302] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5af9d316295641a78bca907d964a26bd; no voters:
I20260927 03:27:07.101425 22196 leader_election.cc:288] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:27:07.101472 22199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:27:07.101531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 22128
I20260927 03:27:07.101642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:27:07.101701 22196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:27:07.101817 392 external_mini_cluster.cc:1428] Setting key c9d0db445093bed79ff5ba575b0936ad
I20260927 03:27:07.101904 22199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [term 1 LEADER]: Becoming Leader. State: Replica: 5af9d316295641a78bca907d964a26bd, State: Running, Role: LEADER
I20260927 03:27:07.102013 22199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af9d316295641a78bca907d964a26bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41577 } }
I20260927 03:27:07.102365 22199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5af9d316295641a78bca907d964a26bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af9d316295641a78bca907d964a26bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41577 } } }
I20260927 03:27:07.102509 22199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:07.102669 22199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5af9d316295641a78bca907d964a26bd. Latest consensus state: current_term: 1 leader_uuid: "5af9d316295641a78bca907d964a26bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af9d316295641a78bca907d964a26bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41577 } } }
I20260927 03:27:07.102730 22210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:27:07.102784 22199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:07.103487 22210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:27:07.104233 22209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:07.102459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34419 (local address 127.0.98.62:41577)
0927 03:27:07.102611 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:07.102622 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:07.102659 (+ 37us) server_negotiation.cc:407] Connection header received
0927 03:27:07.102791 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:07.102800 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:07.102865 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:07.103295 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:07.103743 (+ 448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:07.103747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:07.103747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:27:07.103751 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:07.103796 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:27:07.103810 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:07.103986 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:07.104015 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 03:27:07.104056 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260927 03:27:07.105233 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:27:07.105314 22210 catalog_manager.cc:1347] Generated new cluster ID: cb5a1af3ebc5421980df61efa3452381
I20260927 03:27:07.105352 22210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:27:07.107584 22209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:07.106547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51167 (local address 127.0.98.62:41577)
0927 03:27:07.106582 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:07.106584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:07.106701 (+ 117us) server_negotiation.cc:407] Connection header received
0927 03:27:07.106758 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:07.106758 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:07.106764 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:07.106808 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:07.107359 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:07.107360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:07.107360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:27:07.107364 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:07.107377 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:27:07.107381 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:07.107426 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:07.107496 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 03:27:07.107510 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:27:07.125689 22210 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:27:07.125910 22210 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:27:07.130177 22210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5af9d316295641a78bca907d964a26bd: Generated new TSK 0
I20260927 03:27:07.130393 22210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:27:07.140175 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 22128
2026-09-27T03:27:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:27:07 dist-test-slave-gw2v krb5kdc[22227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:27:07 dist-test-slave-gw2v krb5kdc[22227](info): set up 2 sockets
Sep 27 03:27:07 dist-test-slave-gw2v krb5kdc[22227](info): commencing operation
krb5kdc: starting...
W20260927 03:27:08.102806 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:27:08 dist-test-slave-gw2v krb5kdc[22227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:27:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:27:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:27:08.217327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39639
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:27:08.298358 22240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:08.298596 22240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:08.298630 22240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:08.299922 22240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:08.300040 22240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:08.300060 22240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:08.300076 22240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:08.300091 22240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:08.301731 22240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:08.302002 22240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:08.302232 22240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:08.307277 22248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:08.307332 22246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:08.307277 22245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:08.307524 22240 server_base.cc:1034] running on GCE node
I20260927 03:27:08.307824 22240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:08.308157 22240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:08.309335 22240 hybrid_clock.cc:648] HybridClock initialized: now 1790479628309313 us; error 39 us; skew 500 ppm
Sep 27 03:27:08 dist-test-slave-gw2v krb5kdc[22227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:08.312049 22240 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:27:08.312476 22240 webserver.cc:460] Webserver started at http://127.0.98.62:44801/ using document root <none> and password file <none>
I20260927 03:27:08.312691 22240 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:08.312736 22240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:08.312870 22240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:08.313920 22240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "9a06ec34dcdf44d2bbe2a19d13f6faff"
format_stamp: "Formatted at 2026-09-27 03:27:08 on dist-test-slave-gw2v"
server_key: "f8efed14990b7822c238941258884cd2"
server_key_iv: "0ff3d1b0ac52a535929b40c853c9abae"
server_key_version: "encryptionkey@0"
I20260927 03:27:08.314244 22240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "9a06ec34dcdf44d2bbe2a19d13f6faff"
format_stamp: "Formatted at 2026-09-27 03:27:08 on dist-test-slave-gw2v"
server_key: "f8efed14990b7822c238941258884cd2"
server_key_iv: "0ff3d1b0ac52a535929b40c853c9abae"
server_key_version: "encryptionkey@0"
I20260927 03:27:08.315500 22240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:27:08.316138 22255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:08.316283 22240 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:27:08.316366 22240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "9a06ec34dcdf44d2bbe2a19d13f6faff"
format_stamp: "Formatted at 2026-09-27 03:27:08 on dist-test-slave-gw2v"
server_key: "f8efed14990b7822c238941258884cd2"
server_key_iv: "0ff3d1b0ac52a535929b40c853c9abae"
server_key_version: "encryptionkey@0"
I20260927 03:27:08.316425 22240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:08.324988 22240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:08.327097 22240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:08.327231 22240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:08.330842 22240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39639
I20260927 03:27:08.330889 22307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39639 every 8 connection(s)
I20260927 03:27:08.331212 22240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:27:08.331943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 22240
I20260927 03:27:08.332011 22308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:08.332048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:27:08.332206 392 external_mini_cluster.cc:1428] Setting key d2c5c73eb3215208e812be3872a266f8
I20260927 03:27:08.335209 22308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff: Bootstrap starting.
I20260927 03:27:08.335898 22308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:08.336150 22308 log.cc:824] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:08.336817 22308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff: No bootstrap required, opened a new log
Sep 27 03:27:08 dist-test-slave-gw2v krb5kdc[22227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:27:08.339293 22308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a06ec34dcdf44d2bbe2a19d13f6faff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39639 } }
I20260927 03:27:08.339502 22308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:27:08.339550 22308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a06ec34dcdf44d2bbe2a19d13f6faff, State: Initialized, Role: FOLLOWER
I20260927 03:27:08.339692 22308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a06ec34dcdf44d2bbe2a19d13f6faff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39639 } }
I20260927 03:27:08.339772 22308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:27:08.339824 22308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:27:08.339907 22308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:08.340639 22308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a06ec34dcdf44d2bbe2a19d13f6faff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39639 } }
I20260927 03:27:08.340754 22308 leader_election.cc:302] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a06ec34dcdf44d2bbe2a19d13f6faff; no voters:
I20260927 03:27:08.340912 22308 leader_election.cc:288] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:27:08.340960 22313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:27:08.341148 22313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [term 1 LEADER]: Becoming Leader. State: Replica: 9a06ec34dcdf44d2bbe2a19d13f6faff, State: Running, Role: LEADER
I20260927 03:27:08.341177 22308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:27:08.341282 22313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a06ec34dcdf44d2bbe2a19d13f6faff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39639 } }
I20260927 03:27:08.341636 22314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a06ec34dcdf44d2bbe2a19d13f6faff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a06ec34dcdf44d2bbe2a19d13f6faff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39639 } } }
I20260927 03:27:08.341707 22314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:08.341918 22315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a06ec34dcdf44d2bbe2a19d13f6faff. Latest consensus state: current_term: 1 leader_uuid: "9a06ec34dcdf44d2bbe2a19d13f6faff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a06ec34dcdf44d2bbe2a19d13f6faff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39639 } } }
I20260927 03:27:08.342162 22315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:08.342322 22319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:27:08.342787 22319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:27:08.343812 22311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:08.332801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40033 (local address 127.0.98.62:39639)
0927 03:27:08.332986 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:08.332995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:08.333028 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:27:08.333498 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:08.333508 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:08.333569 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:08.333715 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:08.334301 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:08.335636 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:08.336492 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:08.336933 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:08.337015 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:08.339931 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:08.339935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:08.339936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:08.339947 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:08.341894 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:08.342287 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:08.342290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:08.342291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:08.342399 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:08.342603 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:08.342605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:08.342605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:08.342936 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:08.343092 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:08.343495 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 03:27:08.343626 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260927 03:27:08.344291 22319 catalog_manager.cc:1347] Generated new cluster ID: cf6c9cad8a1b4553937530532e2187ed
I20260927 03:27:08.344338 22319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:27:08.347551 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:27:08.358592 22319 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:27:08.359336 22319 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:27:08.360677 22311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:08.351185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43137 (local address 127.0.98.62:39639)
0927 03:27:08.351229 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:08.351231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:08.355011 (+ 3780us) server_negotiation.cc:407] Connection header received
0927 03:27:08.355073 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:08.355074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:08.355084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:08.355131 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:08.355669 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:08.356163 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:08.356879 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:08.357101 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:08.357164 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:08.358363 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:08.358365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:08.358365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:08.358372 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:08.359670 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:08.359983 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:08.359984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:08.359985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:08.360035 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:08.360201 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:08.360201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:08.360202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:08.360311 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:08.360416 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:08.360512 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 03:27:08.360590 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:27:08.366653 22319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a06ec34dcdf44d2bbe2a19d13f6faff: Generated new TSK 0
I20260927 03:27:08.366888 22319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:27:08.368999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 22240
2026-09-27T03:27:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1231 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:27:08 dist-test-slave-gw2v krb5kdc[22339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:27:08 dist-test-slave-gw2v krb5kdc[22339](info): set up 2 sockets
Sep 27 03:27:08 dist-test-slave-gw2v krb5kdc[22339](info): commencing operation
krb5kdc: starting...
W20260927 03:27:09.330750 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:27:09 dist-test-slave-gw2v krb5kdc[22339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:27:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:27:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:27:09.449784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38883
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:27:09.529249 22352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:09.529466 22352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:09.529497 22352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:09.530659 22352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:09.530709 22352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:09.530726 22352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:09.530742 22352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:09.530757 22352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:09.532311 22352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:09.532562 22352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:09.532788 22352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:09.537751 22357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:09.537768 22360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:09.537760 22358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:09.537961 22352 server_base.cc:1034] running on GCE node
I20260927 03:27:09.538297 22352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:09.538595 22352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:09.539781 22352 hybrid_clock.cc:648] HybridClock initialized: now 1790479629539760 us; error 37 us; skew 500 ppm
Sep 27 03:27:09 dist-test-slave-gw2v krb5kdc[22339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:09.547984 22352 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:27:09.548440 22352 webserver.cc:460] Webserver started at http://127.0.98.62:46061/ using document root <none> and password file <none>
I20260927 03:27:09.548658 22352 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:09.548717 22352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:09.548846 22352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:09.549925 22352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "404246850ccc495c8a8c3159275c823c"
format_stamp: "Formatted at 2026-09-27 03:27:09 on dist-test-slave-gw2v"
server_key: "96b6bdb4cf587f82de6a2ebaab310688"
server_key_iv: "f9481bd43bbe4db40fd10addc6c03812"
server_key_version: "encryptionkey@0"
I20260927 03:27:09.550246 22352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "404246850ccc495c8a8c3159275c823c"
format_stamp: "Formatted at 2026-09-27 03:27:09 on dist-test-slave-gw2v"
server_key: "96b6bdb4cf587f82de6a2ebaab310688"
server_key_iv: "f9481bd43bbe4db40fd10addc6c03812"
server_key_version: "encryptionkey@0"
I20260927 03:27:09.551546 22352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:27:09.552330 22367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:09.552481 22352 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:27:09.552568 22352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "404246850ccc495c8a8c3159275c823c"
format_stamp: "Formatted at 2026-09-27 03:27:09 on dist-test-slave-gw2v"
server_key: "96b6bdb4cf587f82de6a2ebaab310688"
server_key_iv: "f9481bd43bbe4db40fd10addc6c03812"
server_key_version: "encryptionkey@0"
I20260927 03:27:09.552630 22352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:09.573047 22352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:09.575304 22352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:09.575448 22352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:09.579298 22352 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38883
I20260927 03:27:09.579371 22419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38883 every 8 connection(s)
I20260927 03:27:09.579694 22352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:27:09.580585 22420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:09.582883 22420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c: Bootstrap starting.
I20260927 03:27:09.583447 22420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:09.583714 22420 log.cc:824] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:09.584419 22420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c: No bootstrap required, opened a new log
I20260927 03:27:09.584810 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 22352
I20260927 03:27:09.584918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:27:09.585088 392 external_mini_cluster.cc:1428] Setting key bc9c979ee57255a8f4400490811b2ca2
I20260927 03:27:09.586428 22420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404246850ccc495c8a8c3159275c823c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38883 } }
I20260927 03:27:09.586573 22420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:27:09.586615 22420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404246850ccc495c8a8c3159275c823c, State: Initialized, Role: FOLLOWER
I20260927 03:27:09.586758 22420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404246850ccc495c8a8c3159275c823c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38883 } }
I20260927 03:27:09.586858 22420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:27:09.586899 22420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:27:09.586948 22420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:09.587476 22420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404246850ccc495c8a8c3159275c823c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38883 } }
I20260927 03:27:09.587570 22420 leader_election.cc:302] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 404246850ccc495c8a8c3159275c823c; no voters:
I20260927 03:27:09.587735 22420 leader_election.cc:288] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:27:09.587787 22425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:27:09.587988 22425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [term 1 LEADER]: Becoming Leader. State: Replica: 404246850ccc495c8a8c3159275c823c, State: Running, Role: LEADER
I20260927 03:27:09.587991 22420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:27:09.588140 22425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404246850ccc495c8a8c3159275c823c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38883 } }
I20260927 03:27:09.588491 22427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 404246850ccc495c8a8c3159275c823c. Latest consensus state: current_term: 1 leader_uuid: "404246850ccc495c8a8c3159275c823c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404246850ccc495c8a8c3159275c823c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38883 } } }
I20260927 03:27:09.588689 22427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:09.588632 22426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "404246850ccc495c8a8c3159275c823c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404246850ccc495c8a8c3159275c823c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38883 } } }
I20260927 03:27:09.589044 22426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:09.589202 22434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:27:09.589723 22434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:27:09.591355 22434 catalog_manager.cc:1347] Generated new cluster ID: 357a5e8708d948c8a099528346262288
I20260927 03:27:09.591418 22434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:27:09.591596 22424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:09.585653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48579 (local address 127.0.98.62:38883)
0927 03:27:09.585808 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:09.585816 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:09.585840 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:27:09.586400 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:09.586407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:09.586476 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:09.586607 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:09.586853 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:09.588268 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:09.589368 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:09.589895 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:09.590004 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:09.590779 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:09.590783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:09.590783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:27:09.591027 (+ 244us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:09.591064 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:27:09.591076 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:09.591133 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:09.591221 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 03:27:09.591284 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260927 03:27:09.592438 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:27:09.597394 22424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:09.593926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59583 (local address 127.0.98.62:38883)
0927 03:27:09.593968 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:09.593970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:09.594187 (+ 217us) server_negotiation.cc:407] Connection header received
0927 03:27:09.594249 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:09.594249 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:09.594259 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:09.594303 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:09.594426 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:09.595149 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:09.596154 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:09.596438 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:09.596491 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:09.597164 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:09.597165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:09.597165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:27:09.597170 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:09.597183 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:27:09.597187 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:09.597226 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:09.597290 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 03:27:09.597313 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:27:09.601467 22434 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:27:09.601990 22434 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:27:09.606611 22434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 404246850ccc495c8a8c3159275c823c: Generated new TSK 0
I20260927 03:27:09.606804 22434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:27:09.613200 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 22352
2026-09-27T03:27:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1241 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:27:09 dist-test-slave-gw2v krb5kdc[22451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:27:09 dist-test-slave-gw2v krb5kdc[22451](info): set up 2 sockets
Sep 27 03:27:09 dist-test-slave-gw2v krb5kdc[22451](info): commencing operation
krb5kdc: starting...
W20260927 03:27:10.578843 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:27:10 dist-test-slave-gw2v krb5kdc[22451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:27:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:27:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:27:10.697351 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40923
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:27:10.776027 22464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:10.776227 22464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:10.776248 22464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:10.777379 22464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:10.777421 22464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:10.777434 22464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:10.777446 22464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:10.777470 22464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:10.778964 22464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:10.779184 22464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:10.779404 22464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:10.784175 22469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:10.784281 22464 server_base.cc:1034] running on GCE node
W20260927 03:27:10.784238 22470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:10.784222 22472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:10.784644 22464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:10.784910 22464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:10.786053 22464 hybrid_clock.cc:648] HybridClock initialized: now 1790479630786027 us; error 37 us; skew 500 ppm
Sep 27 03:27:10 dist-test-slave-gw2v krb5kdc[22451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:10.788482 22464 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:27:10.788887 22464 webserver.cc:460] Webserver started at http://127.0.98.62:36049/ using document root <none> and password file <none>
I20260927 03:27:10.789076 22464 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:10.789117 22464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:10.789215 22464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:10.790246 22464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "28949eadf8a84a288f5271f51d3f29d5"
format_stamp: "Formatted at 2026-09-27 03:27:10 on dist-test-slave-gw2v"
server_key: "77861cef4fee4ccf733edb15ec1d77e7"
server_key_iv: "fb1e96790b8fed63da571054875abccc"
server_key_version: "encryptionkey@0"
I20260927 03:27:10.790545 22464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "28949eadf8a84a288f5271f51d3f29d5"
format_stamp: "Formatted at 2026-09-27 03:27:10 on dist-test-slave-gw2v"
server_key: "77861cef4fee4ccf733edb15ec1d77e7"
server_key_iv: "fb1e96790b8fed63da571054875abccc"
server_key_version: "encryptionkey@0"
I20260927 03:27:10.791865 22464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:27:10.792547 22479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:10.792721 22464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:27:10.792805 22464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "28949eadf8a84a288f5271f51d3f29d5"
format_stamp: "Formatted at 2026-09-27 03:27:10 on dist-test-slave-gw2v"
server_key: "77861cef4fee4ccf733edb15ec1d77e7"
server_key_iv: "fb1e96790b8fed63da571054875abccc"
server_key_version: "encryptionkey@0"
I20260927 03:27:10.792872 22464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:10.796200 22464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:10.797969 22464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:10.798095 22464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:10.801854 22464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40923
I20260927 03:27:10.801893 22531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40923 every 8 connection(s)
I20260927 03:27:10.802204 22464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:27:10.803174 22532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:10.805540 22532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5: Bootstrap starting.
I20260927 03:27:10.806131 22532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:10.806411 22532 log.cc:824] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:10.807157 22532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5: No bootstrap required, opened a new log
I20260927 03:27:10.808790 22532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28949eadf8a84a288f5271f51d3f29d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40923 } }
I20260927 03:27:10.808940 22532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:27:10.808962 22532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28949eadf8a84a288f5271f51d3f29d5, State: Initialized, Role: FOLLOWER
I20260927 03:27:10.809109 22532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28949eadf8a84a288f5271f51d3f29d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40923 } }
I20260927 03:27:10.809181 22532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:27:10.809202 22532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:27:10.809230 22532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:10.809798 22532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28949eadf8a84a288f5271f51d3f29d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40923 } }
I20260927 03:27:10.809897 22532 leader_election.cc:302] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28949eadf8a84a288f5271f51d3f29d5; no voters:
I20260927 03:27:10.810074 22532 leader_election.cc:288] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:27:10.810127 22535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:27:10.810318 22535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [term 1 LEADER]: Becoming Leader. State: Replica: 28949eadf8a84a288f5271f51d3f29d5, State: Running, Role: LEADER
I20260927 03:27:10.810335 22532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:27:10.810487 22535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28949eadf8a84a288f5271f51d3f29d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40923 } }
I20260927 03:27:10.810905 22536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28949eadf8a84a288f5271f51d3f29d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28949eadf8a84a288f5271f51d3f29d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40923 } } }
I20260927 03:27:10.811036 22537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28949eadf8a84a288f5271f51d3f29d5. Latest consensus state: current_term: 1 leader_uuid: "28949eadf8a84a288f5271f51d3f29d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28949eadf8a84a288f5271f51d3f29d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40923 } } }
I20260927 03:27:10.811046 22536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:10.811137 22537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5 [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:10.811529 22544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:27:10.811878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 22464
I20260927 03:27:10.811973 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:27:10.812023 22544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:27:10.812161 392 external_mini_cluster.cc:1428] Setting key 5dac36c565c466e55914f13fc6375dcd
I20260927 03:27:10.813555 22544 catalog_manager.cc:1347] Generated new cluster ID: dd5e6ff39e064d4f892df8c83d7b1b91
I20260927 03:27:10.813611 22544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:27:10.820760 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:10.812625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60453 (local address 127.0.98.62:40923)
0927 03:27:10.812807 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:10.812819 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:10.819126 (+ 6307us) server_negotiation.cc:407] Connection header received
0927 03:27:10.819274 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:10.819282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:10.819315 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:10.819464 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:10.820286 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:10.820290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:10.820290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:27:10.820294 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:10.820350 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:27:10.820365 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:10.820425 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:10.820507 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 03:27:10.820549 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260927 03:27:10.821679 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:27:10.824205 22544 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:27:10.824380 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:10.823218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36583 (local address 127.0.98.62:40923)
0927 03:27:10.823258 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:10.823260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:10.823364 (+ 104us) server_negotiation.cc:407] Connection header received
0927 03:27:10.823420 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:10.823421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:10.823426 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:10.823477 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:10.824077 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:10.824078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:10.824078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:27:10.824083 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:10.824099 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:27:10.824103 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:10.824157 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:10.824322 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 03:27:10.824337 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:27:10.824445 22544 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:27:10.831698 22544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28949eadf8a84a288f5271f51d3f29d5: Generated new TSK 0
I20260927 03:27:10.831887 22544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:27:10.834352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 22464
2026-09-27T03:27:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1221 ms)
[----------] 18 tests from AuthTokenIssuingTest (22207 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:27:10 dist-test-slave-gw2v krb5kdc[22563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:27:10 dist-test-slave-gw2v krb5kdc[22563](info): set up 2 sockets
Sep 27 03:27:10 dist-test-slave-gw2v krb5kdc[22563](info): commencing operation
krb5kdc: starting...
W20260927 03:27:11.794902 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:27:11 dist-test-slave-gw2v krb5kdc[22563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:27:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:27:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:27:11.911847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33997
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33997,127.0.98.61:43269,127.0.98.60:35859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:27:11.991566 22576 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:27:11.991801 22576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:11.991833 22576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:11.991860 22576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:11.993038 22576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:11.993081 22576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:11.993100 22576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:11.993116 22576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:11.993129 22576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:11.994578 22576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33997,127.0.98.61:43269,127.0.98.60:35859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:11.994834 22576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:11.995043 22576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:11.999987 22581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:11.999970 22582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:11.999970 22584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:12.000301 22576 server_base.cc:1034] running on GCE node
I20260927 03:27:12.000537 22576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:12.000828 22576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:12.001986 22576 hybrid_clock.cc:648] HybridClock initialized: now 1790479632001972 us; error 28 us; skew 500 ppm
Sep 27 03:27:11 dist-test-slave-gw2v krb5kdc[22563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:12.004474 22576 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:27:12.004923 22576 webserver.cc:460] Webserver started at http://127.0.98.62:35331/ using document root <none> and password file <none>
I20260927 03:27:12.005155 22576 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:12.005210 22576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:12.005333 22576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:12.006500 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "da0092bedf814a71a4030704404f6bee"
format_stamp: "Formatted at 2026-09-27 03:27:11 on dist-test-slave-gw2v"
server_key: "0fa7e8b9009e9d3d6c1f2180300371ed"
server_key_iv: "dfd6be451f37c918102d37935d805b69"
server_key_version: "encryptionkey@0"
I20260927 03:27:12.006999 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "da0092bedf814a71a4030704404f6bee"
format_stamp: "Formatted at 2026-09-27 03:27:11 on dist-test-slave-gw2v"
server_key: "0fa7e8b9009e9d3d6c1f2180300371ed"
server_key_iv: "dfd6be451f37c918102d37935d805b69"
server_key_version: "encryptionkey@0"
I20260927 03:27:12.008327 22576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:27:12.009008 22591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:12.009147 22576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:27:12.009279 22576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "da0092bedf814a71a4030704404f6bee"
format_stamp: "Formatted at 2026-09-27 03:27:11 on dist-test-slave-gw2v"
server_key: "0fa7e8b9009e9d3d6c1f2180300371ed"
server_key_iv: "dfd6be451f37c918102d37935d805b69"
server_key_version: "encryptionkey@0"
I20260927 03:27:12.009349 22576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:12.019862 22576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:12.021919 22576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:12.022115 22576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:12.025967 22576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33997
I20260927 03:27:12.026000 22643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33997 every 8 connection(s)
I20260927 03:27:12.026326 22576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:27:12.027300 22644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:12.028833 22644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:27:12 dist-test-slave-gw2v krb5kdc[22563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:27:12.036386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 22576
I20260927 03:27:12.036464 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:27:12.036607 392 external_mini_cluster.cc:1428] Setting key 258dc2932ab4b71746350baa1a295bc7
I20260927 03:27:12.038194 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.029489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33997 (local address 127.0.0.1:34952)
0927 03:27:12.029627 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.029660 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:12.030084 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:12.030363 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:12.030366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:12.030595 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:12.030745 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.030749 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.031856 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.031857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.032549 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.032551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.033054 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.033055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.033103 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.033531 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:12.033539 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:12.034987 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:12.036668 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.036675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.036677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.036904 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.037179 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.037179 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.037180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.037283 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.037598 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:12.037602 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:12.037668 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 03:27:12.037756 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 03:27:12.037930 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260927 03:27:12.038208 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.029650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34952 (local address 127.0.98.62:33997)
0927 03:27:12.029780 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.029784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.029810 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:27:12.030240 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.030243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.030262 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.030357 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.030875 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.031771 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.032649 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.032973 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.033060 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.035059 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:12.035066 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:12.035066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:12.035073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:12.036599 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.036976 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.036978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.036979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.037055 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.037338 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.037339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.037339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.037542 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:12.037618 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.037826 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.037990 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":104,"thread_start_us":46,"threads_started":1}
I20260927 03:27:12.038918 22644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:27:12.039222 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.039131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43269 (local address 127.0.0.1:37084)
0927 03:27:12.039169 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.039206 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 03:27:12.039338 22592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43269: connect: Connection refused (error 111)
I20260927 03:27:12.039678 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.039608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43269 (local address 127.0.0.1:37098)
0927 03:27:12.039641 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.039663 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 03:27:12.039762 22644 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43269: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43269: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 03:27:12.061403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43269
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33997,127.0.98.61:43269,127.0.98.60:35859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:27:12.088980 22644 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } attempt: 1
I20260927 03:27:12.089471 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.089347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43269 (local address 127.0.0.1:37108)
0927 03:27:12.089408 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.089442 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 03:27:12.089855 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.089760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43269 (local address 127.0.0.1:37114)
0927 03:27:12.089816 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.089840 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 03:27:12.089954 22644 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43269: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43269: connect: Connection refused (error 111)
W20260927 03:27:12.141057 22651 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:27:12.141309 22651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:12.141335 22651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:12.141358 22651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:12.142519 22651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:12.142562 22651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:12.142581 22651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:12.142598 22651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:12.142612 22651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:12.144109 22651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33997,127.0.98.61:43269,127.0.98.60:35859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:12.144361 22651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:12.144562 22651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:12.149125 22656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:12.149217 22651 server_base.cc:1034] running on GCE node
W20260927 03:27:12.149209 22657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:12.149121 22659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:12.149631 22651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:12.149914 22651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:12.151052 22651 hybrid_clock.cc:648] HybridClock initialized: now 1790479632151036 us; error 27 us; skew 500 ppm
Sep 27 03:27:12 dist-test-slave-gw2v krb5kdc[22563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:12.153435 22651 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 03:27:12.153859 22651 webserver.cc:460] Webserver started at http://127.0.98.61:35017/ using document root <none> and password file <none>
I20260927 03:27:12.154029 22651 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:12.154069 22651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:12.154152 22651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:12.155207 22651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/data/instance:
uuid: "74533edbb811444fbc41bb312697fe5e"
format_stamp: "Formatted at 2026-09-27 03:27:12 on dist-test-slave-gw2v"
server_key: "c4555c9552393102501bee355fefec89"
server_key_iv: "63ddd2a490bd1ac32d29ff2065ad6aba"
server_key_version: "encryptionkey@0"
I20260927 03:27:12.155489 22651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/wal/instance:
uuid: "74533edbb811444fbc41bb312697fe5e"
format_stamp: "Formatted at 2026-09-27 03:27:12 on dist-test-slave-gw2v"
server_key: "c4555c9552393102501bee355fefec89"
server_key_iv: "63ddd2a490bd1ac32d29ff2065ad6aba"
server_key_version: "encryptionkey@0"
I20260927 03:27:12.156697 22651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:27:12.157531 22666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:12.157719 22651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:27:12.157810 22651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/wal
uuid: "74533edbb811444fbc41bb312697fe5e"
format_stamp: "Formatted at 2026-09-27 03:27:12 on dist-test-slave-gw2v"
server_key: "c4555c9552393102501bee355fefec89"
server_key_iv: "63ddd2a490bd1ac32d29ff2065ad6aba"
server_key_version: "encryptionkey@0"
I20260927 03:27:12.157869 22651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:12.158082 22644 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } attempt: 2
I20260927 03:27:12.158423 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.158312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43269 (local address 127.0.0.1:37118)
0927 03:27:12.158363 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.158395 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 03:27:12.158775 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.158651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43269 (local address 127.0.0.1:37126)
0927 03:27:12.158705 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.158745 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:27:12.158901 22644 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43269: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43269: connect: Connection refused (error 111)
I20260927 03:27:12.169948 22651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:12.172258 22651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:12.172375 22651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:12.176232 22651 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43269
I20260927 03:27:12.176266 22718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43269 every 8 connection(s)
I20260927 03:27:12.176581 22651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/data/info.pb
I20260927 03:27:12.177491 22719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:12.179035 22719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:27:12 dist-test-slave-gw2v krb5kdc[22563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:27:12.186228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 22651
I20260927 03:27:12.186319 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-1/wal/instance
I20260927 03:27:12.186512 392 external_mini_cluster.cc:1428] Setting key ee7f76bf78131b287a31c41f75c5c6a3
I20260927 03:27:12.187744 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.179730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34964 (local address 127.0.98.62:33997)
0927 03:27:12.179822 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.179824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.179972 (+ 148us) server_negotiation.cc:407] Connection header received
0927 03:27:12.180503 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.180504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.180519 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.180576 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.181204 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.181696 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.182641 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.182887 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.182947 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.185073 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:12.185074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:12.185075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:12.185082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:12.186443 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.186851 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.186852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.186853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.186921 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.187133 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.187134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.187134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.187255 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:12.187311 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.187569 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.187654 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 03:27:12.187927 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.179662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33997 (local address 127.0.0.1:34964)
0927 03:27:12.179856 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.179895 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:12.180394 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:12.180674 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:12.180677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:12.180916 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:12.181117 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.181121 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.181781 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.181782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.182564 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.182567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.182969 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.182970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.183092 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.183621 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:12.183630 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:12.184986 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:12.186506 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.186514 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.186517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.186765 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.186975 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.186975 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.186976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.187080 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.187308 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:12.187315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:12.187454 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 03:27:12.187543 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 03:27:12.187710 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20260927 03:27:12.188468 22719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:27:12 dist-test-slave-gw2v krb5kdc[22563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 03:27:12.195588 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.188718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43269 (local address 127.0.0.1:37130)
0927 03:27:12.188759 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.188768 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:12.188804 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:12.189083 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:12.189084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:12.189225 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:12.189279 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.189281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.190492 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.190493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.191048 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.191050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.191666 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.191667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.191735 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.192091 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:12.192095 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:12.193177 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:12.194712 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.194714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.194714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.194892 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.195047 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.195048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.195048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.195091 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.195334 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:12.195335 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:12.195376 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 03:27:12.195429 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:27:12.195505 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 03:27:12.195655 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.188759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.0.98.61:43269)
0927 03:27:12.188911 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.188916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.188938 (+ 22us) server_negotiation.cc:407] Connection header received
0927 03:27:12.188959 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.188961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.188974 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.189055 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.189354 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.190413 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.191115 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.191596 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.191667 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.193250 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:12.193258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:12.193259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:12.193265 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:12.194627 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.194953 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.194955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.194955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.195003 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.195122 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.195123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.195123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.195266 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:12.195320 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.195455 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.195551 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 03:27:12.196271 22719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:27:12.196568 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.196471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35859 (local address 127.0.0.1:40658)
0927 03:27:12.196510 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.196552 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 03:27:12.196699 22668 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
I20260927 03:27:12.197070 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.196995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35859 (local address 127.0.0.1:40666)
0927 03:27:12.197033 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.197055 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 03:27:12.197155 22719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35859: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 03:27:12.209985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35859
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33997,127.0.98.61:43269,127.0.98.60:35859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:27:12.246416 22719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } attempt: 1
I20260927 03:27:12.246963 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.246809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35859 (local address 127.0.0.1:40678)
0927 03:27:12.246883 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.246931 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 03:27:12.247362 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.247226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35859 (local address 127.0.0.1:40686)
0927 03:27:12.247273 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.247299 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:27:12.247462 22719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35859: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
I20260927 03:27:12.250046 22644 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } attempt: 3
Sep 27 03:27:12 dist-test-slave-gw2v krb5kdc[22563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:27:12.256570 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.250378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37136 (local address 127.0.98.61:43269)
0927 03:27:12.250431 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.250433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.250447 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:27:12.250516 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.250516 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.250528 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.250574 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.250898 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.251418 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.252106 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.252329 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.252385 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.254354 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:12.254356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:12.254356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:12.254363 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:12.255626 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.255967 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.255968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.255968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.256014 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.256177 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.256178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.256178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.256269 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:12.256311 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.256451 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.256509 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:27:12.256616 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.250316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43269 (local address 127.0.0.1:37136)
0927 03:27:12.250395 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.250406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:12.250465 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:12.250576 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:12.250577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:12.250729 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:12.250790 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.250792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.251512 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.251513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.252034 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.252035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.252388 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.252389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.252475 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.252881 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:12.252886 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:12.254275 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:12.255694 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.255695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.255695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.255904 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.256064 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.256065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.256065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.256125 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.256317 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:12.256317 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:12.256368 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 03:27:12.256426 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:27:12.256521 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 03:27:12.256976 22644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:27:12.257295 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.257189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35859 (local address 127.0.0.1:40700)
0927 03:27:12.257234 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.257270 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 03:27:12.257606 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.257527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35859 (local address 127.0.0.1:40708)
0927 03:27:12.257566 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.257588 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:27:12.257688 22644 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35859: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
I20260927 03:27:12.288864 22644 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } attempt: 1
I20260927 03:27:12.289342 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.289183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35859 (local address 127.0.0.1:40712)
0927 03:27:12.289257 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.289305 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 03:27:12.289746 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.289656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35859 (local address 127.0.0.1:40714)
0927 03:27:12.289696 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.289723 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 03:27:12.289870 22644 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35859: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
W20260927 03:27:12.290771 22726 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:27:12.291002 22726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:12.291018 22726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:12.291041 22726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:12.292220 22726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:12.292258 22726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:12.292271 22726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:12.292282 22726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:12.292294 22726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:12.293751 22726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33997,127.0.98.61:43269,127.0.98.60:35859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:12.293977 22726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:12.294185 22726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:12.298743 22732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:12.298885 22731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:12.299011 22726 server_base.cc:1034] running on GCE node
W20260927 03:27:12.298748 22734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:12.299366 22726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:12.299649 22726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:12.300815 22726 hybrid_clock.cc:648] HybridClock initialized: now 1790479632300802 us; error 30 us; skew 500 ppm
Sep 27 03:27:12 dist-test-slave-gw2v krb5kdc[22563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:12.303345 22726 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 03:27:12.303766 22726 webserver.cc:460] Webserver started at http://127.0.98.60:43023/ using document root <none> and password file <none>
I20260927 03:27:12.303968 22726 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:12.304015 22726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:12.304155 22726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:12.305200 22726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/data/instance:
uuid: "efaf3bdae56b4394b82186cff78c9544"
format_stamp: "Formatted at 2026-09-27 03:27:12 on dist-test-slave-gw2v"
server_key: "e5e5a7b6b8d326a9a1bad2dc447f56bb"
server_key_iv: "e1d6f42d26bb7598d992c023b3c3b84c"
server_key_version: "encryptionkey@0"
I20260927 03:27:12.305529 22726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/wal/instance:
uuid: "efaf3bdae56b4394b82186cff78c9544"
format_stamp: "Formatted at 2026-09-27 03:27:12 on dist-test-slave-gw2v"
server_key: "e5e5a7b6b8d326a9a1bad2dc447f56bb"
server_key_iv: "e1d6f42d26bb7598d992c023b3c3b84c"
server_key_version: "encryptionkey@0"
I20260927 03:27:12.306702 22726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:27:12.307379 22741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:12.307523 22726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:12.307610 22726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/wal
uuid: "efaf3bdae56b4394b82186cff78c9544"
format_stamp: "Formatted at 2026-09-27 03:27:12 on dist-test-slave-gw2v"
server_key: "e5e5a7b6b8d326a9a1bad2dc447f56bb"
server_key_iv: "e1d6f42d26bb7598d992c023b3c3b84c"
server_key_version: "encryptionkey@0"
I20260927 03:27:12.307669 22726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:12.315600 22719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } attempt: 2
I20260927 03:27:12.316035 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.315898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35859 (local address 127.0.0.1:40728)
0927 03:27:12.315954 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.315998 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 03:27:12.316430 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.316323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35859 (local address 127.0.0.1:40740)
0927 03:27:12.316364 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.316404 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 03:27:12.316509 22719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35859: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35859: connect: Connection refused (error 111)
I20260927 03:27:12.343761 22726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:12.345947 22726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:12.346081 22726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:12.349910 22726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35859
I20260927 03:27:12.349962 22793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35859 every 8 connection(s)
I20260927 03:27:12.350293 22726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/data/info.pb
I20260927 03:27:12.351281 22794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:12.352824 22794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:27:12.355127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 22726
I20260927 03:27:12.355225 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479369225731-392-0/minicluster-data/master-2/wal/instance
I20260927 03:27:12.355415 392 external_mini_cluster.cc:1428] Setting key cfcf8d9c92f90c838b90f8f66e557c91
Sep 27 03:27:12 dist-test-slave-gw2v krb5kdc[22563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 03:27:12 dist-test-slave-gw2v krb5kdc[22563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:27:12.362102 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.353511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34976 (local address 127.0.98.62:33997)
0927 03:27:12.353600 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.353602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.353715 (+ 113us) server_negotiation.cc:407] Connection header received
0927 03:27:12.354203 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.354205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.354217 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.354275 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.354883 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.355560 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.356892 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.357088 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.357153 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.359226 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:12.359227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:12.359227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:12.359233 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:12.360679 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.361154 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.361156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.361156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.361211 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.361437 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.361438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.361439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.361556 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:12.361612 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.361976 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.362038 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 03:27:12.362308 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.353438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33997 (local address 127.0.0.1:34976)
0927 03:27:12.353597 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.353642 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:12.354098 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:12.354333 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:12.354337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:12.354602 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:12.354792 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.354797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.355649 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.355649 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.356826 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.356829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.357179 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.357180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.357307 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.357824 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:12.357832 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:12.359150 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:12.360752 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.360758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.360759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.361081 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.361276 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.361277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.361277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.361380 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.361649 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:12.361655 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:12.361807 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 03:27:12.361949 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 03:27:12.362089 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260927 03:27:12.362864 22794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:27:12.363653 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.355762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.0.98.60:35859)
0927 03:27:12.355911 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.355915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.355960 (+ 45us) server_negotiation.cc:407] Connection header received
0927 03:27:12.356023 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.356024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.356039 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.356100 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.356769 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.357530 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.358273 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.358598 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.358655 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.360840 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:12.360846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:12.360846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:12.360852 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:12.362469 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.362892 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.362895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.362895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.362946 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.363174 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.363175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.363176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.363332 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:12.363468 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.363523 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.363586 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260927 03:27:12.365015 22644 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } attempt: 2
Sep 27 03:27:12 dist-test-slave-gw2v krb5kdc[22563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 03:27:12 dist-test-slave-gw2v krb5kdc[22563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:27:12.369653 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.363116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43269 (local address 127.0.0.1:37138)
0927 03:27:12.363155 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.363163 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:12.363191 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:12.363297 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:12.363298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:12.363422 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:12.363452 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.363454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.364170 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.364171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.364802 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.364804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.365150 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.365151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.365208 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.365684 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:12.365688 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:12.366917 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:12.368597 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.368598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.368599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.368810 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.369019 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.369020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.369020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.369082 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.369360 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:12.369361 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:12.369407 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 03:27:12.369489 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 03:27:12.369568 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 03:27:12.369700 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.363155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37138 (local address 127.0.98.61:43269)
0927 03:27:12.363194 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.363195 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.363219 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:27:12.363241 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.363241 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.363249 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.363283 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.363516 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.364108 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.364869 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.365090 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.365152 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.367004 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:12.367006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:12.367006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:12.367012 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:12.368524 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.368888 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.368888 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.368889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.368944 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.369152 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.369152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.369153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.369279 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:12.369346 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.369526 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.369611 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 27 03:27:12 dist-test-slave-gw2v krb5kdc[22563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:27:12.370013 22794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:27:12.373541 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.364823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34982 (local address 127.0.98.62:33997)
0927 03:27:12.365205 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.365207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.365212 (+ 5us) server_negotiation.cc:407] Connection header received
0927 03:27:12.365226 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.365227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.365233 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.365283 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.365794 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.366234 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.367197 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.367449 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.367514 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.370517 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:12.370518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:12.370518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:12.370524 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:12.372191 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.372566 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.372567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.372567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.372623 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.373033 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.373034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.373035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.373162 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:12.373348 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.373386 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.373454 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365}
Sep 27 03:27:12 dist-test-slave-gw2v krb5kdc[22563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:27:12.374410 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.365211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35859 (local address 127.0.0.1:40758)
0927 03:27:12.365292 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.365301 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:12.365336 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:12.365749 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:12.365750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:12.365907 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:12.365952 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.365953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.367043 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.367045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.367686 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.367689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.368158 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.368159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.368213 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.368546 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:12.368551 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:12.370282 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:12.372776 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.372776 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.372777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.372966 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.373768 (+ 802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.373769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.373769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.373845 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.374107 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:12.374109 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:12.374166 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 03:27:12.374211 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:27:12.374319 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":1}
I20260927 03:27:12.374709 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.365298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.0.98.60:35859)
0927 03:27:12.365331 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.365332 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.365338 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:27:12.365650 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.365651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.365660 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.365714 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.366067 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.366922 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.367779 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.368070 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.368148 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.370359 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:12.370360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:12.370361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:12.370367 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:12.372693 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.373639 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.373640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.373640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.373692 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.373905 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.373906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.373907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.374028 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:12.374509 (+ 481us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.374565 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.374629 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:27:12.376667 22644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee: Bootstrap starting.
I20260927 03:27:12.377213 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.370249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35859 (local address 127.0.0.1:40764)
0927 03:27:12.370391 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.370401 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:12.370431 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:12.370551 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:12.370552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:12.370667 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:12.370691 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.370692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.371401 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.371402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.372137 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.372140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.372584 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.372585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.372642 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.372971 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:12.372974 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:12.374263 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:12.376324 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.376325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.376325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.376474 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.376632 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.376632 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.376633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.376682 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.376925 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:12.376926 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:12.376964 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 03:27:12.377021 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:27:12.377115 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260927 03:27:12.377315 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.370279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.0.98.60:35859)
0927 03:27:12.370428 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.370430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.370437 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:27:12.370479 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.370480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.370490 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.370549 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.370748 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.371321 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.372218 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.372510 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.372572 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.374427 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:12.374429 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:12.374429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:12.374436 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:12.376243 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.376530 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.376531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.376532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.376581 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.376729 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.376729 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.376729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.376861 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:12.376923 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.377050 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.377162 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260927 03:27:12.377418 22644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:12.377737 22644 log.cc:824] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:12.378580 22644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee: No bootstrap required, opened a new log
I20260927 03:27:12.378618 22794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544: Bootstrap starting.
Sep 27 03:27:12 dist-test-slave-gw2v krb5kdc[22563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:27:12.379349 22794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:12.379644 22794 log.cc:824] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:12.380474 22794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544: No bootstrap required, opened a new log
I20260927 03:27:12.381016 22644 raft_consensus.cc:348] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } }
I20260927 03:27:12.381171 22644 raft_consensus.cc:729] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da0092bedf814a71a4030704404f6bee, State: Initialized, Role: FOLLOWER
I20260927 03:27:12.381286 22644 consensus_queue.cc:260] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } }
I20260927 03:27:12.381625 22802 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } } }
I20260927 03:27:12.381673 22644 sys_catalog.cc:564] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:27:12.381762 22802 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:27:12.382226 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.374672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.0.98.61:43269)
0927 03:27:12.374717 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.374719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.374885 (+ 166us) server_negotiation.cc:407] Connection header received
0927 03:27:12.374987 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.374989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.374999 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.375053 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.375609 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.376259 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.377138 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.377404 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.377460 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.379744 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:12.379745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:12.379745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:12.379753 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:12.381027 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.381361 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.381362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.381362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.381419 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.381612 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.381612 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.381613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.381741 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:12.381886 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.382073 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.382141 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:27:12.382484 22794 raft_consensus.cc:348] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } }
I20260927 03:27:12.382645 22794 raft_consensus.cc:729] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efaf3bdae56b4394b82186cff78c9544, State: Initialized, Role: FOLLOWER
I20260927 03:27:12.382776 22794 consensus_queue.cc:260] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } }
I20260927 03:27:12.383097 22808 sys_catalog.cc:455] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } } }
I20260927 03:27:12.383164 22794 sys_catalog.cc:564] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:27:12.383237 22808 sys_catalog.cc:458] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:27:12.383628 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 03:27:12.384290 22815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:27:12.384341 22815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:27:12.384580 22618 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "da0092bedf814a71a4030704404f6bee"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34982
I20260927 03:27:12.384691 22618 raft_consensus.cc:482] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 03:27:12.384733 22618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:12.385464 22618 raft_consensus.cc:504] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } }
I20260927 03:27:12.385663 22618 leader_election.cc:288] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [CANDIDATE]: Term 1 election: Requested vote from peers 74533edbb811444fbc41bb312697fe5e (127.0.98.61:43269), efaf3bdae56b4394b82186cff78c9544 (127.0.98.60:35859)
I20260927 03:27:12.384989 22576 master_runner.cc:186] Error getting master registration for 127.0.98.61:43269: OK, instance_id { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" instance_seqno: 1790479632175175 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:27:12.387379 22576 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:27:12.387609 22768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da0092bedf814a71a4030704404f6bee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "efaf3bdae56b4394b82186cff78c9544"
I20260927 03:27:12.387717 22768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:12.389029 22768 raft_consensus.cc:2454] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da0092bedf814a71a4030704404f6bee in term 1.
I20260927 03:27:12.389250 22593 leader_election.cc:302] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da0092bedf814a71a4030704404f6bee, efaf3bdae56b4394b82186cff78c9544; no voters:
I20260927 03:27:12.389371 22802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:27:12.389534 22802 raft_consensus.cc:686] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [term 1 LEADER]: Becoming Leader. State: Replica: da0092bedf814a71a4030704404f6bee, State: Running, Role: LEADER
I20260927 03:27:12.389613 22802 consensus_queue.cc:237] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } }
I20260927 03:27:12.390012 22802 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [sys.catalog]: SysCatalogTable state changed. Reason: New leader da0092bedf814a71a4030704404f6bee. Latest consensus state: current_term: 1 leader_uuid: "da0092bedf814a71a4030704404f6bee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } } }
I20260927 03:27:12.390091 22802 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:12.391993 22726 master_runner.cc:186] Error getting master registration for 127.0.98.61:43269: OK, instance_id { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" instance_seqno: 1790479632175175 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:27:12.394326 22726 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:27:12.395195 22836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:27:12.395498 22836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:27:12.396098 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.387318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.0.98.60:35859)
0927 03:27:12.387455 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.387456 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.387548 (+ 92us) server_negotiation.cc:407] Connection header received
0927 03:27:12.387631 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.387632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.387642 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.387669 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.388272 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.389029 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.390039 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.390394 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.390432 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.392022 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:12.392024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:12.392024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:12.392030 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:12.393536 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.395253 (+ 1717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.395254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.395254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.395315 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.395503 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.395504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.395504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.395636 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:12.395717 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.395928 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.396010 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260927 03:27:12.396843 22826 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:27:12.396944 22826 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:27:12.397168 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.387358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.0.98.62:33997)
0927 03:27:12.387400 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.387401 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.387409 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:27:12.387439 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.387440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.387448 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.387493 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.388580 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.389245 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.390584 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.390882 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.390933 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.393367 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:12.393369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:12.393370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:12.393377 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:12.394958 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.396133 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.396134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.396135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.396209 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.396479 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.396479 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.396480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.396616 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:12.396683 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.397005 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.397082 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:27:12.402478 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.387406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37154 (local address 127.0.98.61:43269)
0927 03:27:12.387540 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.387542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.387548 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:27:12.391970 (+ 4422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.391971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.391985 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.392024 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.394893 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.395620 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.398375 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.398643 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.398700 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.399822 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:12.399823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:12.399823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:12.399831 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:12.401451 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.401769 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.401770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.401771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.401825 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.401989 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.401989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.401990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.402105 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:12.402203 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.402308 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.402388 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260927 03:27:12.403811 22768 raft_consensus.cc:1261] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544 [term 1 FOLLOWER]: Refusing update from remote peer da0092bedf814a71a4030704404f6bee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:27:12.404433 22834 consensus_queue.cc:1035] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:27:12.405823 22808 sys_catalog.cc:455] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da0092bedf814a71a4030704404f6bee. Latest consensus state: current_term: 1 leader_uuid: "da0092bedf814a71a4030704404f6bee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } } }
I20260927 03:27:12.405915 22808 sys_catalog.cc:458] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:27:12.406971 22834 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da0092bedf814a71a4030704404f6bee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } } }
I20260927 03:27:12.407086 22834 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:12.407605 22808 sys_catalog.cc:455] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da0092bedf814a71a4030704404f6bee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } } }
I20260927 03:27:12.407691 22808 sys_catalog.cc:458] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:27:12.407977 22839 mvcc.cc:205] Tried to move back new op lower bound from 7333804574322274304 to 7333804574267748352. Current Snapshot: MvccSnapshot[applied={T|T < 7333804574322274304}]
I20260927 03:27:12.408164 22719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } attempt: 3
I20260927 03:27:12.408543 22836 catalog_manager.cc:1347] Generated new cluster ID: 97f9b08cf3a64475b46013641e9c4f0c
I20260927 03:27:12.408595 22836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:27:12 dist-test-slave-gw2v krb5kdc[22563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:27:12.417296 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.408502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.0.98.60:35859)
0927 03:27:12.408542 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.408543 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.408555 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:27:12.408644 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.408645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.408656 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.408698 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.409082 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.409832 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.410874 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.411176 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.411238 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.414115 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:12.414116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:12.414117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:12.414124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:12.415882 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.416465 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.416466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.416466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.416536 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:12.416744 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:12.416745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:12.416745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:12.416887 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:12.416939 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.417118 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.417204 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:27:12.417341 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.408470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35859 (local address 127.0.0.1:40792)
0927 03:27:12.408522 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:12.408532 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:12.408570 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:12.408741 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:12.408742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:12.408942 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:12.408993 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.408995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.409891 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.409892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.410755 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:12.410759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.411244 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.411245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:12.411319 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.412321 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:12.412328 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:12.414019 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:12.416048 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.416050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.416050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.416351 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.416604 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:12.416605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:12.416605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:12.416677 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:12.416946 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:12.416947 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:12.416998 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 03:27:12.417074 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 03:27:12.417228 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 03:27:12.419128 22719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e: Bootstrap starting.
I20260927 03:27:12.419972 22719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:12.420364 22719 log.cc:824] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:12.421058 22836 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:27:12.421849 22836 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:27:12.427608 22836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee: Generated new TSK 0
I20260927 03:27:12.428004 22836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:27:12.428663 22719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e: No bootstrap required, opened a new log
I20260927 03:27:12.430670 22719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } }
I20260927 03:27:12.430846 22719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74533edbb811444fbc41bb312697fe5e, State: Initialized, Role: FOLLOWER
I20260927 03:27:12.430926 22719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } }
I20260927 03:27:12.431268 22719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:27:12.431354 22692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:12.432344 22692 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e [term 1 FOLLOWER]: Refusing update from remote peer da0092bedf814a71a4030704404f6bee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:27:12.431293 22693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da0092bedf814a71a4030704404f6bee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74533edbb811444fbc41bb312697fe5e"
I20260927 03:27:12.433552 22802 consensus_queue.cc:1035] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:27:12.433230 22693 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da0092bedf814a71a4030704404f6bee in term 1.
I20260927 03:27:12.437847 22857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader da0092bedf814a71a4030704404f6bee. Latest consensus state: current_term: 1 leader_uuid: "da0092bedf814a71a4030704404f6bee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } } }
I20260927 03:27:12.437938 22857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:27:12.437989 22857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da0092bedf814a71a4030704404f6bee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } } }
I20260927 03:27:12.438032 22857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:27:12.431232 22844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } } }
I20260927 03:27:12.438299 22844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:27:12.440040 22855 catalog_manager.cc:1259] Loaded cluster ID: 97f9b08cf3a64475b46013641e9c4f0c
I20260927 03:27:12.440086 22855 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e: loading cluster ID for follower catalog manager: success
I20260927 03:27:12.441159 22855 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e: acquiring CA information for follower catalog manager: success
I20260927 03:27:12.441465 22855 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 74533edbb811444fbc41bb312697fe5e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 03:27:12.460196 22647 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:12.458502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.0.98.62:33997)
0927 03:27:12.458550 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.458552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.459503 (+ 951us) server_negotiation.cc:407] Connection header received
0927 03:27:12.459607 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.459609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.459619 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.459650 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.460123 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35008: BlockingRecv error: recv got EOF from 127.0.0.1:35008 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 03:27:12.461051 22799 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:12.458979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40804 (local address 127.0.98.60:35859)
0927 03:27:12.459027 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.459029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.460461 (+ 1432us) server_negotiation.cc:407] Connection header received
0927 03:27:12.460553 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.460555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.460565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.460593 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.460974 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40804: BlockingRecv error: recv got EOF from 127.0.0.1:40804 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 03:27:12.461809 22723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:12.459237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.0.98.61:43269)
0927 03:27:12.459284 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.459286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.461282 (+ 1996us) server_negotiation.cc:407] Connection header received
0927 03:27:12.461370 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.461371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.461395 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.461424 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.461717 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37164: BlockingRecv error: recv got EOF from 127.0.0.1:37164 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:27:12.467105 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.463447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37168 (local address 127.0.98.61:43269)
0927 03:27:12.463482 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.463484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.463704 (+ 220us) server_negotiation.cc:407] Connection header received
0927 03:27:12.463805 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.463806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.463850 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.463894 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:12.464112 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.464919 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.466090 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.466391 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.466428 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.466646 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:12.466787 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:12.466864 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.466974 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.466995 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:27:12.470449 22799 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:12.469606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.0.98.60:35859)
0927 03:27:12.469648 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.469649 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.469886 (+ 237us) server_negotiation.cc:407] Connection header received
0927 03:27:12.469986 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.469988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.470009 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.470051 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.470395 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40810: BlockingRecv error: recv got EOF from 127.0.0.1:40810 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:27:12.475823 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.472712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.0.98.61:43269)
0927 03:27:12.472747 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.472749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.472952 (+ 203us) server_negotiation.cc:407] Connection header received
0927 03:27:12.473054 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.473055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.473064 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.473090 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:12.473244 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.473983 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.475095 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.475384 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.475420 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.475594 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:12.475626 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:12.475648 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.475734 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.475751 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 03:27:12.478902 22799 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:12.478099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40812 (local address 127.0.98.60:35859)
0927 03:27:12.478135 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.478136 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.478376 (+ 240us) server_negotiation.cc:407] Connection header received
0927 03:27:12.478467 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.478467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.478478 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.478503 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.478845 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40812: BlockingRecv error: recv got EOF from 127.0.0.1:40812 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:27:12.485287 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.482007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.0.98.61:43269)
0927 03:27:12.482064 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.482065 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.482265 (+ 200us) server_negotiation.cc:407] Connection header received
0927 03:27:12.482351 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.482352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.482362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.482386 (+ 24us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:12.482588 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.483366 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.484449 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.484739 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.484784 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.485010 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:12.485057 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:12.485086 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.485192 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.485212 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260927 03:27:12.488404 22799 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:12.487579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.0.98.60:35859)
0927 03:27:12.487620 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.487622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.487872 (+ 250us) server_negotiation.cc:407] Connection header received
0927 03:27:12.487964 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.487965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.487977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.488005 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.488353 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40818: BlockingRecv error: recv got EOF from 127.0.0.1:40818 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:27:12.496738 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.493567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.0.98.61:43269)
0927 03:27:12.493606 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.493608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.493828 (+ 220us) server_negotiation.cc:407] Connection header received
0927 03:27:12.493924 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.493925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.493934 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.493964 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:12.494151 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.494881 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.495934 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.496226 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.496272 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.496478 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:12.496512 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:12.496539 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.496642 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.496660 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:27:12.499986 22799 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:12.499100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.0.98.60:35859)
0927 03:27:12.499141 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.499143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.499423 (+ 280us) server_negotiation.cc:407] Connection header received
0927 03:27:12.499523 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.499525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.499537 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.499566 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.499928 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40832: BlockingRecv error: recv got EOF from 127.0.0.1:40832 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:27:12.513201 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.509372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.0.98.61:43269)
0927 03:27:12.509429 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.509431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.509744 (+ 313us) server_negotiation.cc:407] Connection header received
0927 03:27:12.509846 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.509848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.509859 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.509887 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:12.510131 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.510934 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.512255 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.512586 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.512634 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.512871 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:12.512908 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:12.512956 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.513095 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.513116 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:27:12.516440 22799 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:12.515543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.0.98.60:35859)
0927 03:27:12.515587 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.515588 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.515847 (+ 259us) server_negotiation.cc:407] Connection header received
0927 03:27:12.515943 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.515944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.515958 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.515987 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.516366 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40838: BlockingRecv error: recv got EOF from 127.0.0.1:40838 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:27:12.537351 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.533855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37222 (local address 127.0.98.61:43269)
0927 03:27:12.533900 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.533902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.534185 (+ 283us) server_negotiation.cc:407] Connection header received
0927 03:27:12.534289 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.534290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.534301 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.534330 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:12.534521 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.535313 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.536486 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.536773 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.536822 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.537073 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:12.537110 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:12.537139 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.537244 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.537267 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:27:12.540638 22799 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:12.539771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40848 (local address 127.0.98.60:35859)
0927 03:27:12.539811 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.539813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.540080 (+ 267us) server_negotiation.cc:407] Connection header received
0927 03:27:12.540177 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.540178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.540190 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.540218 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.540579 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40848: BlockingRecv error: recv got EOF from 127.0.0.1:40848 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:27:12.576189 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.573862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.0.98.61:43269)
0927 03:27:12.573910 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.573913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.574059 (+ 146us) server_negotiation.cc:407] Connection header received
0927 03:27:12.574124 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.574125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.574135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.574182 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:12.574303 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.574802 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.575597 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.575777 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.575831 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.575963 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:12.575990 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:12.576030 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.576089 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.576107 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:27:12.578572 22799 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:12.577814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.0.98.60:35859)
0927 03:27:12.577857 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.577858 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.578097 (+ 239us) server_negotiation.cc:407] Connection header received
0927 03:27:12.578155 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.578156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.578169 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.578214 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.578523 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40860: BlockingRecv error: recv got EOF from 127.0.0.1:40860 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:27:12.645843 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.643713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37248 (local address 127.0.98.61:43269)
0927 03:27:12.643778 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.643780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.643870 (+ 90us) server_negotiation.cc:407] Connection header received
0927 03:27:12.643941 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.643941 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.643950 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.644010 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:12.644114 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.644598 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.645366 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.645541 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.645587 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.645681 (+ 94us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:12.645706 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:12.645737 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.645768 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.645782 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 03:27:12.648029 22799 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:12.647548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40866 (local address 127.0.98.60:35859)
0927 03:27:12.647634 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.647636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.647644 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:27:12.647665 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.647666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.647676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.647721 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.647977 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40866: BlockingRecv error: recv got EOF from 127.0.0.1:40866 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 03:27:12.779506 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:12.777256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.0.98.61:43269)
0927 03:27:12.777303 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.777305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.777374 (+ 69us) server_negotiation.cc:407] Connection header received
0927 03:27:12.777438 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.777439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.777449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.777485 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:12.777608 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.778086 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.778940 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:12.779123 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:12.779179 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:12.779328 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:12.779356 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:12.779388 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:12.779430 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 03:27:12.779446 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 03:27:12.781912 22799 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:12.781206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.0.98.60:35859)
0927 03:27:12.781285 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:12.781287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:12.781327 (+ 40us) server_negotiation.cc:407] Connection header received
0927 03:27:12.781392 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:12.781393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:12.781404 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:12.781452 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:12.781848 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40868: BlockingRecv error: recv got EOF from 127.0.0.1:40868 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 03:27:13.000237 22963 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da0092bedf814a71a4030704404f6bee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0092bedf814a71a4030704404f6bee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "74533edbb811444fbc41bb312697fe5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43269 } } peers { permanent_uuid: "efaf3bdae56b4394b82186cff78c9544" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35859 } } }
I20260927 03:27:13.000388 22963 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0092bedf814a71a4030704404f6bee [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:13.041332 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:13.039060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37260 (local address 127.0.98.61:43269)
0927 03:27:13.039119 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:13.039121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:13.039168 (+ 47us) server_negotiation.cc:407] Connection header received
0927 03:27:13.039233 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:13.039234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:13.039243 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:13.039284 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:13.039396 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:13.039901 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:13.040662 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:13.040854 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:13.040900 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:13.041090 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:13.041117 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:13.041152 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:13.041246 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 03:27:13.041263 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 03:27:13.043666 22799 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:13.043084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40880 (local address 127.0.98.60:35859)
0927 03:27:13.043141 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:13.043143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:13.043152 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:27:13.043202 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:13.043203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:13.043214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:13.043274 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:13.043605 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40880: BlockingRecv error: recv got EOF from 127.0.0.1:40880 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:27:13.397488 22826 catalog_manager.cc:1259] Loaded cluster ID: 97f9b08cf3a64475b46013641e9c4f0c
I20260927 03:27:13.397575 22826 catalog_manager.cc:1552] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544: loading cluster ID for follower catalog manager: success
I20260927 03:27:13.398403 22826 catalog_manager.cc:1574] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544: acquiring CA information for follower catalog manager: success
I20260927 03:27:13.398681 22826 catalog_manager.cc:1602] T 00000000000000000000000000000000 P efaf3bdae56b4394b82186cff78c9544: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:27:13.559095 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:13.556699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.0.98.61:43269)
0927 03:27:13.556821 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:13.556823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:13.556833 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:27:13.556910 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:13.556911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:13.556924 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:13.556981 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:13.557131 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:13.557656 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:13.558408 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:13.558623 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:13.558678 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:13.558806 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:13.558874 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:13.558915 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:13.558980 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:27:13.559005 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":51,"threads_started":1}
I20260927 03:27:13.562777 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:13.560567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.0.98.60:35859)
0927 03:27:13.560662 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:13.560664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:13.560715 (+ 51us) server_negotiation.cc:407] Connection header received
0927 03:27:13.560792 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:13.560793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:13.560844 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:13.560897 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:13.560975 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:13.561485 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:13.562197 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:13.562385 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:13.562432 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:13.562524 (+ 92us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:13.562633 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:13.562686 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:13.562698 (+ 12us) server_negotiation.cc:299] Negotiation successful
0927 03:27:13.562715 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":48,"threads_started":1}
I20260927 03:27:13.566946 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:13.564440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.0.98.60:35859)
0927 03:27:13.564476 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:13.564478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:13.564484 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:27:13.564525 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:13.564526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:13.564534 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:13.564571 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:13.564740 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:13.565429 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:13.566152 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:13.566401 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:13.566459 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:13.566580 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:13.566609 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:13.566655 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:13.566850 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 03:27:13.566872 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:27:13.567378 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:13.564263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.0.98.62:33997)
0927 03:27:13.564513 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:13.564515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:13.564523 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:27:13.564572 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:13.564573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:13.564624 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:13.564698 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:13.564770 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:13.565257 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:13.566264 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:13.566548 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:13.566601 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:13.566931 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:13.567022 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:13.567096 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:13.567270 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 03:27:13.567297 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":48,"threads_started":1}
I20260927 03:27:13.567418 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:13.564355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.0.98.61:43269)
0927 03:27:13.564411 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:13.564413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:13.564421 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:27:13.564446 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:13.564447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:13.564510 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:13.564560 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:13.564646 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:13.565385 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:13.566356 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:13.566607 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:13.566655 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:13.566770 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:13.566801 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:13.567314 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:13.567331 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 03:27:13.567355 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:27:13.568997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 22576
I20260927 03:27:13.573753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 22651
I20260927 03:27:13.578073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 22726
2026-09-27T03:27:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2745 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:27:13 dist-test-slave-gw2v krb5kdc[22997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:27:13 dist-test-slave-gw2v krb5kdc[22997](info): set up 2 sockets
Sep 27 03:27:13 dist-test-slave-gw2v krb5kdc[22997](info): commencing operation
krb5kdc: starting...
W20260927 03:27:14.526770 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/test-user.keytab.
Sep 27 03:27:14 dist-test-slave-gw2v krb5kdc[22997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:27:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:27:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:27:14.641539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41769
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41769,127.0.98.61:45765,127.0.98.60:46691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:27:14.722193 23010 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:27:14.722440 23010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:14.722460 23010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:14.722482 23010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:14.723699 23010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:14.723747 23010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:14.723763 23010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:14.723776 23010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:14.723789 23010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:14.725319 23010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41769,127.0.98.61:45765,127.0.98.60:46691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:14.725571 23010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:14.725807 23010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:14.730749 23016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:14.730806 23018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:14.730785 23015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:14.730945 23010 server_base.cc:1034] running on GCE node
I20260927 03:27:14.731232 23010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:14.731530 23010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:14.732692 23010 hybrid_clock.cc:648] HybridClock initialized: now 1790479634732667 us; error 44 us; skew 500 ppm
Sep 27 03:27:14 dist-test-slave-gw2v krb5kdc[22997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:14.738793 23010 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:27:14.739307 23010 webserver.cc:460] Webserver started at http://127.0.98.62:34811/ using document root <none> and password file <none>
I20260927 03:27:14.739542 23010 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:14.739590 23010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:14.739723 23010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:14.740825 23010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/data/instance:
uuid: "533e2a6a7b994c0ea8b7d2f385201c77"
format_stamp: "Formatted at 2026-09-27 03:27:14 on dist-test-slave-gw2v"
server_key: "008daeca678a8b26f9fd8ee0757e15ea"
server_key_iv: "ce3110cb02ebf52f35934600eca04da3"
server_key_version: "encryptionkey@0"
I20260927 03:27:14.741165 23010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/wal/instance:
uuid: "533e2a6a7b994c0ea8b7d2f385201c77"
format_stamp: "Formatted at 2026-09-27 03:27:14 on dist-test-slave-gw2v"
server_key: "008daeca678a8b26f9fd8ee0757e15ea"
server_key_iv: "ce3110cb02ebf52f35934600eca04da3"
server_key_version: "encryptionkey@0"
I20260927 03:27:14.742609 23010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:27:14.743474 23025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:14.743664 23010 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:27:14.743753 23010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/wal
uuid: "533e2a6a7b994c0ea8b7d2f385201c77"
format_stamp: "Formatted at 2026-09-27 03:27:14 on dist-test-slave-gw2v"
server_key: "008daeca678a8b26f9fd8ee0757e15ea"
server_key_iv: "ce3110cb02ebf52f35934600eca04da3"
server_key_version: "encryptionkey@0"
I20260927 03:27:14.743829 23010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:14.757328 23010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:14.759260 23010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:14.759380 23010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:14.763279 23010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41769
I20260927 03:27:14.763324 23077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41769 every 8 connection(s)
I20260927 03:27:14.763664 23010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:27:14.764617 23078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:14.766078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 23010
I20260927 03:27:14.766175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-0/wal/instance
I20260927 03:27:14.766371 392 external_mini_cluster.cc:1428] Setting key 2aa784e04da0a10cd3d7a4ca5f543fc0
I20260927 03:27:14.766446 23078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:27:14 dist-test-slave-gw2v krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 03:27:14.777318 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.767277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41769 (local address 127.0.0.1:46174)
0927 03:27:14.767436 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.767480 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:14.768072 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:14.768386 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:14.768389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:14.768740 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:14.768972 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:14.768978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.770100 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.770101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:14.771223 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:14.771227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.771744 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.771745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:14.771821 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:14.772362 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:14.772373 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:14.774123 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:14.775868 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:14.775876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:14.775878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:14.776086 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:14.776287 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:14.776288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:14.776288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:14.776391 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:14.776817 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:14.776823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:14.776886 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 03:27:14.776962 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 03:27:14.777093 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260927 03:27:14.777323 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.767466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.0.98.62:41769)
0927 03:27:14.767620 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:14.767626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:14.767672 (+ 46us) server_negotiation.cc:407] Connection header received
0927 03:27:14.768250 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:14.768252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:14.768271 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:14.768361 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:14.769094 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.770007 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.771305 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.771640 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.771732 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:14.774229 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:14.774237 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:14.774237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:14.774245 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:14.775783 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:14.776150 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:14.776153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:14.776154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:14.776235 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:14.776468 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:14.776469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:14.776470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:14.776736 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:14.776839 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:14.777025 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 03:27:14.777175 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260927 03:27:14.778066 23078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:27:14.778383 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.778280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45765 (local address 127.0.0.1:33080)
0927 03:27:14.778320 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.778363 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 03:27:14.778529 23028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45765: connect: Connection refused (error 111)
I20260927 03:27:14.778946 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.778877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45765 (local address 127.0.0.1:33094)
0927 03:27:14.778908 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.778930 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260927 03:27:14.779048 23078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45765: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45765: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 03:27:14.793059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45765
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41769,127.0.98.61:45765,127.0.98.60:46691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:27:14.828286 23078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } attempt: 1
I20260927 03:27:14.828823 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.828656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45765 (local address 127.0.0.1:33100)
0927 03:27:14.828725 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.828785 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 03:27:14.829180 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.829097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45765 (local address 127.0.0.1:33116)
0927 03:27:14.829133 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.829160 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 03:27:14.829309 23078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45765: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45765: connect: Connection refused (error 111)
W20260927 03:27:14.873482 23085 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:27:14.873693 23085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:14.873716 23085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:14.873735 23085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:14.874864 23085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:14.874904 23085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:14.874917 23085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:14.874929 23085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:14.874940 23085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:14.876645 23085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41769,127.0.98.61:45765,127.0.98.60:46691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:14.876902 23085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:14.877087 23085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:14.881677 23090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:14.881721 23091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:14.881747 23093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:14.881879 23085 server_base.cc:1034] running on GCE node
I20260927 03:27:14.882135 23085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:14.882413 23085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:14.883574 23085 hybrid_clock.cc:648] HybridClock initialized: now 1790479634883561 us; error 27 us; skew 500 ppm
Sep 27 03:27:14 dist-test-slave-gw2v krb5kdc[22997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:14.886044 23085 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 03:27:14.886518 23085 webserver.cc:460] Webserver started at http://127.0.98.61:39719/ using document root <none> and password file <none>
I20260927 03:27:14.886714 23085 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:14.886760 23085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:14.886909 23085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:14.887943 23085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/data/instance:
uuid: "501881b8e7e04538a2527b5a4ce2e848"
format_stamp: "Formatted at 2026-09-27 03:27:14 on dist-test-slave-gw2v"
server_key: "9b783f5ae90b334d084d3b09f6c07d69"
server_key_iv: "75b7ac233b6647daaed14b9129c911a2"
server_key_version: "encryptionkey@0"
I20260927 03:27:14.888281 23085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/wal/instance:
uuid: "501881b8e7e04538a2527b5a4ce2e848"
format_stamp: "Formatted at 2026-09-27 03:27:14 on dist-test-slave-gw2v"
server_key: "9b783f5ae90b334d084d3b09f6c07d69"
server_key_iv: "75b7ac233b6647daaed14b9129c911a2"
server_key_version: "encryptionkey@0"
I20260927 03:27:14.889530 23085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:27:14.890201 23100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:14.890358 23085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:27:14.890439 23085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/wal
uuid: "501881b8e7e04538a2527b5a4ce2e848"
format_stamp: "Formatted at 2026-09-27 03:27:14 on dist-test-slave-gw2v"
server_key: "9b783f5ae90b334d084d3b09f6c07d69"
server_key_iv: "75b7ac233b6647daaed14b9129c911a2"
server_key_version: "encryptionkey@0"
I20260927 03:27:14.890501 23085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:14.897452 23078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } attempt: 2
I20260927 03:27:14.897847 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.897734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45765 (local address 127.0.0.1:33128)
0927 03:27:14.897786 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.897820 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 03:27:14.898138 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.898063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45765 (local address 127.0.0.1:33140)
0927 03:27:14.898094 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.898117 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 03:27:14.898213 23078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45765: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45765: connect: Connection refused (error 111)
I20260927 03:27:14.901021 23085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:14.902884 23085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:14.903017 23085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:14.906777 23085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45765
I20260927 03:27:14.906808 23152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45765 every 8 connection(s)
I20260927 03:27:14.907199 23085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/data/info.pb
I20260927 03:27:14.907729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 23085
I20260927 03:27:14.907827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-1/wal/instance
I20260927 03:27:14.908046 392 external_mini_cluster.cc:1428] Setting key b1521570c321196722671123dcea5743
I20260927 03:27:14.908347 23153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:14.910156 23153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 03:27:14 dist-test-slave-gw2v krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:27:14.921914 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.910815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.0.98.62:41769)
0927 03:27:14.910896 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:14.910898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:14.911081 (+ 183us) server_negotiation.cc:407] Connection header received
0927 03:27:14.911656 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:14.911657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:14.911670 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:14.911755 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:14.912568 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.913063 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.914485 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.914670 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.914736 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:14.919200 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:14.919202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:14.919202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:14.919210 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:14.920556 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:14.920986 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:14.920987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:14.920987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:14.921039 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:14.921287 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:14.921288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:14.921288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:14.921390 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:14.921437 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:14.921747 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 03:27:14.921826 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 03:27:14.922154 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.910766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41769 (local address 127.0.0.1:46186)
0927 03:27:14.910948 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.910993 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:14.911558 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:14.911839 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:14.911843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:14.912192 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:14.912468 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:14.912475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.913150 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.913152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:14.914406 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:14.914409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.914754 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.914755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:14.914938 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:14.915447 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:14.915457 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:14.919090 (+ 3633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:14.920627 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:14.920635 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:14.920637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:14.920919 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:14.921091 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:14.921092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:14.921092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:14.921233 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:14.921442 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:14.921447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:14.921617 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 03:27:14.921730 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 03:27:14.921907 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260927 03:27:14.922878 23153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:27:14 dist-test-slave-gw2v krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:27:14.930554 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.923141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45765 (local address 127.0.0.1:33146)
0927 03:27:14.923331 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.923342 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:14.923383 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:14.923503 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:14.923504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:14.923638 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:14.923688 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:14.923689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.924517 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.924518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:14.925012 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:14.925013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.925446 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.925447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:14.925505 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:14.925889 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:14.925893 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:14.928005 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:14.929565 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:14.929566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:14.929566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:14.929764 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:14.929937 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:14.929937 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:14.929937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:14.929984 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:14.930230 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:14.930231 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:14.930268 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 03:27:14.930329 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:27:14.930427 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 03:27:14.930617 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.923206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33146 (local address 127.0.98.61:45765)
0927 03:27:14.923356 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:14.923360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:14.923382 (+ 22us) server_negotiation.cc:407] Connection header received
0927 03:27:14.923429 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:14.923431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:14.923444 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:14.923504 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:14.923753 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.924448 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.925075 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.925387 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.925443 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:14.928078 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:14.928086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:14.928086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:14.928093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:14.929495 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:14.929833 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:14.929836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:14.929836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:14.929885 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:14.930029 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:14.930030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:14.930030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:14.930171 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:14.930243 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:14.930362 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 03:27:14.930494 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260927 03:27:14.931262 23153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:27:14.931586 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.931477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46691 (local address 127.0.0.1:43258)
0927 03:27:14.931512 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.931562 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260927 03:27:14.931710 23101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46691: connect: Connection refused (error 111)
I20260927 03:27:14.932081 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.932004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46691 (local address 127.0.0.1:43264)
0927 03:27:14.932044 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.932066 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:27:14.932165 23153 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46691: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46691: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 03:27:14.933058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzllPpO/build/release/bin/kudu
/tmp/dist-test-taskzllPpO/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46691
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41769,127.0.98.61:45765,127.0.98.60:46691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:27:14.981395 23153 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } attempt: 1
I20260927 03:27:14.981993 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.981807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46691 (local address 127.0.0.1:43276)
0927 03:27:14.981895 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.981955 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 03:27:14.982421 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.982307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46691 (local address 127.0.0.1:43284)
0927 03:27:14.982366 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.982396 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 03:27:14.982518 23153 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46691: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46691: connect: Connection refused (error 111)
I20260927 03:27:14.989358 23078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } attempt: 3
Sep 27 03:27:14 dist-test-slave-gw2v krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:27:14.996480 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.989801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.0.98.61:45765)
0927 03:27:14.989865 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:14.989867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:14.989877 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:27:14.989905 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:14.989905 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:14.989916 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:14.989962 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:14.990255 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.990736 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.991462 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.991666 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.991723 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:14.993733 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:14.993734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:14.993735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:14.993742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:14.995362 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:14.995725 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:14.995725 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:14.995726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:14.995774 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:14.995963 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:14.995963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:14.995964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:14.996077 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:14.996127 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:14.996347 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 03:27:14.996411 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:27:14.996528 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.989731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45765 (local address 127.0.0.1:33152)
0927 03:27:14.989789 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.989801 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:14.989852 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:14.989973 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:14.989974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:14.990122 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:14.990186 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:14.990187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.990853 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.990855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:14.991389 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:14.991391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:14.991726 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:14.991727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:14.991796 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:14.992191 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:14.992195 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:14.993647 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:14.995440 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:14.995441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:14.995442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:14.995653 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:14.995828 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:14.995829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:14.995829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:14.995900 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:14.996133 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:14.996134 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:14.996202 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 03:27:14.996316 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 03:27:14.996427 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:27:14.996942 23078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:27:14.997308 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.997194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46691 (local address 127.0.0.1:43290)
0927 03:27:14.997251 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.997285 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 03:27:14.997622 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:14.997549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46691 (local address 127.0.0.1:43302)
0927 03:27:14.997586 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:14.997607 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 03:27:14.997689 23078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46691: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46691: connect: Connection refused (error 111)
W20260927 03:27:15.015409 23160 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:27:15.015630 23160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:15.015645 23160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:15.015664 23160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:15.016851 23160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:15.016894 23160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:15.016906 23160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:15.016917 23160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:15.016928 23160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:15.018432 23160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41769,127.0.98.61:45765,127.0.98.60:46691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:15.018667 23160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:15.018935 23160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:15.023613 23166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:15.023627 23165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:15.023622 23168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:15.023873 23160 server_base.cc:1034] running on GCE node
I20260927 03:27:15.024109 23160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:15.024394 23160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:15.025540 23160 hybrid_clock.cc:648] HybridClock initialized: now 1790479635025520 us; error 31 us; skew 500 ppm
Sep 27 03:27:15 dist-test-slave-gw2v krb5kdc[22997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:15.028034 23160 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 03:27:15.028494 23160 webserver.cc:460] Webserver started at http://127.0.98.60:39001/ using document root <none> and password file <none>
I20260927 03:27:15.028693 23160 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:15.028734 23160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:15.028853 23160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:15.028823 23078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } attempt: 1
I20260927 03:27:15.029192 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.029064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46691 (local address 127.0.0.1:43310)
0927 03:27:15.029124 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:15.029160 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:27:15.029495 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.029410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46691 (local address 127.0.0.1:43316)
0927 03:27:15.029444 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:15.029472 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 03:27:15.029585 23078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46691: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46691: connect: Connection refused (error 111)
I20260927 03:27:15.030128 23160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/data/instance:
uuid: "45336c89a11a40608dbd4a31cb418367"
format_stamp: "Formatted at 2026-09-27 03:27:15 on dist-test-slave-gw2v"
server_key: "23b32cabf74df4be67121c02979640a3"
server_key_iv: "993c12a3bcf5586273bce04c6b1423e3"
server_key_version: "encryptionkey@0"
I20260927 03:27:15.030484 23160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/wal/instance:
uuid: "45336c89a11a40608dbd4a31cb418367"
format_stamp: "Formatted at 2026-09-27 03:27:15 on dist-test-slave-gw2v"
server_key: "23b32cabf74df4be67121c02979640a3"
server_key_iv: "993c12a3bcf5586273bce04c6b1423e3"
server_key_version: "encryptionkey@0"
I20260927 03:27:15.031855 23160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:27:15.032692 23175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:15.032881 23160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:27:15.032971 23160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/wal
uuid: "45336c89a11a40608dbd4a31cb418367"
format_stamp: "Formatted at 2026-09-27 03:27:15 on dist-test-slave-gw2v"
server_key: "23b32cabf74df4be67121c02979640a3"
server_key_iv: "993c12a3bcf5586273bce04c6b1423e3"
server_key_version: "encryptionkey@0"
I20260927 03:27:15.033036 23160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:15.044925 23160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:15.047137 23160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:15.047279 23160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:15.050648 23153 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } attempt: 2
I20260927 03:27:15.051119 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.050966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46691 (local address 127.0.0.1:43318)
0927 03:27:15.051033 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:15.051078 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 03:27:15.051172 23160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46691
I20260927 03:27:15.051286 23227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46691 every 8 connection(s)
I20260927 03:27:15.051599 23160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/data/info.pb
I20260927 03:27:15.052276 23228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:15.053990 23228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:27:15 dist-test-slave-gw2v krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:27:15.057649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzllPpO/build/release/bin/kudu as pid 23160
I20260927 03:27:15.057744 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzllPpO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479369225731-392-0/minicluster-data/master-2/wal/instance
I20260927 03:27:15.057940 392 external_mini_cluster.cc:1428] Setting key 09990681dd67de944d383628bdbc6a89
Sep 27 03:27:15 dist-test-slave-gw2v krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:27:15.060961 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.051385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46691 (local address 127.0.0.1:43330)
0927 03:27:15.051422 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:15.051433 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:15.051490 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:15.052862 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:15.052863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:15.053026 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:15.053079 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:15.053082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.054159 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.054161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:15.054920 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:15.054924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.055529 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.055530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:15.055577 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.056011 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:15.056015 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:15.057357 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:15.059457 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:15.059458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:15.059459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:15.059667 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:15.059918 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:15.059919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:15.059919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:15.059984 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:15.060624 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:15.060625 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:15.060674 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 03:27:15.060745 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 03:27:15.060853 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 03:27:15.061208 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.051541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.0.98.60:46691)
0927 03:27:15.052146 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.052155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.052181 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:27:15.052665 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.052672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.052724 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.052844 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.053170 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.054062 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.054983 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.055454 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.055560 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.057439 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:15.057444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:15.057445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:15.057455 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:15.059388 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.059748 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.059751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.059752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.059854 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.060035 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.060036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.060036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.060347 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:15.060433 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:15.060830 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 03:27:15.060958 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":506,"threads_started":1}
I20260927 03:27:15.062659 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.054670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.0.98.62:41769)
0927 03:27:15.054721 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.054723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.054913 (+ 190us) server_negotiation.cc:407] Connection header received
0927 03:27:15.054984 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.054985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.054996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.055054 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.055457 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.056100 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.056985 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.057214 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.057281 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.059595 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:15.059596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:15.059597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:15.059604 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:15.061176 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.061923 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.061924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.061925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.061994 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.062166 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.062166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.062167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.062277 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:15.062337 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:15.062496 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 03:27:15.062567 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:27:15.062673 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.054583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41769 (local address 127.0.0.1:46194)
0927 03:27:15.054789 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:15.054849 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:15.054919 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:15.055083 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:15.055085 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:15.055329 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:15.055389 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:15.055391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.056194 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.056195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:15.056903 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:15.056906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.057308 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.057309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:15.057364 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.057856 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:15.057860 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:15.059519 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:15.061638 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:15.061641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:15.061643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:15.061868 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:15.062049 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:15.062050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:15.062050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:15.062113 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:15.062337 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:15.062344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:15.062394 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 03:27:15.062463 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:27:15.062567 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":101,"threads_started":1}
I20260927 03:27:15.063230 23153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848: Bootstrap starting.
Sep 27 03:27:15 dist-test-slave-gw2v krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:27:15.064067 23153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:15.064327 23153 log.cc:824] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:15.064916 23228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:27:15.065176 23153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848: No bootstrap required, opened a new log
I20260927 03:27:15.067288 23153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } }
I20260927 03:27:15.067458 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.058232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.0.98.60:46691)
0927 03:27:15.058357 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.058358 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.058626 (+ 268us) server_negotiation.cc:407] Connection header received
0927 03:27:15.058683 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.058684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.058693 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.058750 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.059364 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.060209 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.061159 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.061445 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.061481 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.064039 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:15.064040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:15.064040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:15.064047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:15.066182 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.066578 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.066579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.066580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.066642 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.066858 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.066859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.066859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.066982 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:15.067110 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:15.067188 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:27:15.067261 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260927 03:27:15.067444 23153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501881b8e7e04538a2527b5a4ce2e848, State: Initialized, Role: FOLLOWER
I20260927 03:27:15.067529 23153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } }
I20260927 03:27:15.067891 23153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 03:27:15 dist-test-slave-gw2v krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:27:15.069957 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } } }
I20260927 03:27:15.070056 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:27:15.070583 23246 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:27:15.070649 23246 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:27:15.072148 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.065196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.0.98.61:45765)
0927 03:27:15.065235 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.065237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.065268 (+ 31us) server_negotiation.cc:407] Connection header received
0927 03:27:15.065329 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.065330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.065338 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.065379 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.065700 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.066161 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.066929 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.067227 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.067290 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.069351 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:15.069352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:15.069353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:15.069360 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:15.070990 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.071350 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.071351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.071351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.071404 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.071593 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.071594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.071594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.071755 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:15.071800 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:15.071991 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 03:27:15.072063 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:27:15.072430 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.065151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45765 (local address 127.0.0.1:33158)
0927 03:27:15.065229 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:15.065239 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:15.065280 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:15.065432 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:15.065433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:15.065577 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:15.065631 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:15.065633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.066224 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.066225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:15.066869 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:15.066871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.067343 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.067344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:15.067410 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.067965 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:15.067970 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:15.069248 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:15.071076 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:15.071077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:15.071078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:15.071270 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:15.071481 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:15.071482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:15.071482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:15.071534 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:15.071844 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:15.071845 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:15.071888 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:27:15.072250 (+ 362us) client_negotiation.cc:240] Negotiation successful
0927 03:27:15.072349 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 03:27:15.070426 23085 master_runner.cc:186] Error getting master registration for 127.0.98.62:41769: OK, instance_id { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" instance_seqno: 1790479634762155 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:27:15.072738 23085 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:27:15.072757 23228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:27:15 dist-test-slave-gw2v krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:27:15.076792 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.068308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.0.98.62:41769)
0927 03:27:15.068376 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.068378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.068387 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:27:15.068407 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.068408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.068417 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.068474 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.069045 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.069740 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.071374 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.071666 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.071705 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.074032 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:15.074033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:15.074034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:15.074040 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:15.075524 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.075875 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.075875 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.075876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.075932 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.076133 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.076135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.076135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.076268 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:15.076411 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:15.076624 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 03:27:15.076698 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 03:27:15 dist-test-slave-gw2v krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:27:15.081229 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.073078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.0.98.60:46691)
0927 03:27:15.073113 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.073115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.074958 (+ 1843us) server_negotiation.cc:407] Connection header received
0927 03:27:15.075054 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.075055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.075069 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.075131 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.075413 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.076021 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.076648 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.076860 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.076917 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.078736 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:15.078738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:15.078738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:15.078745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:15.080035 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.080305 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.080306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.080306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.080350 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.080496 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.080496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.080497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.080624 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:15.080679 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:15.080812 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 03:27:15.080917 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":19}
I20260927 03:27:15.081229 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.073006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46691 (local address 127.0.0.1:43354)
0927 03:27:15.074903 (+ 1897us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:15.074914 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:15.074983 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:15.075139 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:15.075140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:15.075284 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:15.075334 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:15.075336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.076084 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.076084 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:15.076575 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:15.076577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.076961 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.076962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:15.077001 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.077363 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:15.077368 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:15.078653 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:15.080103 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:15.080104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:15.080104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:15.080249 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:15.080399 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:15.080399 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:15.080400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:15.080448 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:15.080679 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:15.080679 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:15.080717 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 03:27:15.080775 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:27:15.080868 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1881}
Sep 27 03:27:15 dist-test-slave-gw2v krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:27:15.082616 23228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367: Bootstrap starting.
I20260927 03:27:15.083300 23228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:15.083588 23228 log.cc:824] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:15.084669 23228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367: No bootstrap required, opened a new log
I20260927 03:27:15.085654 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.077792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33160 (local address 127.0.98.61:45765)
0927 03:27:15.077827 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.077829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.077837 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:27:15.077858 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.077858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.077867 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.077919 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.078490 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.079115 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.080363 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.080638 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.080677 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.083170 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:15.083172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:15.083172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:15.083179 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:15.084543 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.084916 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.084917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.084918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.084977 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.085152 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.085153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.085153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.085280 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:15.085423 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:15.085491 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 03:27:15.085567 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:27:15.086688 23228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } }
I20260927 03:27:15.086889 23228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45336c89a11a40608dbd4a31cb418367, State: Initialized, Role: FOLLOWER
I20260927 03:27:15.086977 23228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } }
I20260927 03:27:15.087062 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:27:15.087291 23228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:27:15.087256 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } } }
I20260927 03:27:15.087363 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:27:15.087962 23052 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "533e2a6a7b994c0ea8b7d2f385201c77"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46200
W20260927 03:27:15.089303 23260 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:27:15.089516 23260 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:27:15.089600 23160 master_runner.cc:186] Error getting master registration for 127.0.98.62:41769: OK, instance_id { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" instance_seqno: 1790479634762155 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:27:15.091908 23160 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:27:15.104743 23078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } attempt: 2
Sep 27 03:27:15 dist-test-slave-gw2v krb5kdc[22997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:27:15.112713 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.105113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46691 (local address 127.0.0.1:43364)
0927 03:27:15.105168 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:15.105180 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:15.105222 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:15.105369 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:15.105370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:15.105554 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:15.105606 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:15.105608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.106510 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.106512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:15.107386 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:15.107389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.107836 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.107837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:15.107897 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.108392 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:15.108397 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:15.109906 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:15.111689 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:15.111690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:15.111691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:15.111898 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:15.112078 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:15.112078 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:15.112079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:15.112151 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:15.112396 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:15.112397 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:15.112444 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 03:27:15.112515 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 03:27:15.112608 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:27:15.113070 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.105193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43364 (local address 127.0.98.60:46691)
0927 03:27:15.105235 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.105237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.105246 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:27:15.105274 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.105274 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.105285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.105358 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.105667 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.106428 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.107459 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.107761 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.107829 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.109985 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:15.109987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:15.109988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:15.109995 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:15.111617 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.111971 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.111972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.111973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.112029 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.112205 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.112205 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.112206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.112338 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:15.112401 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:15.112900 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 03:27:15.112979 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:27:15.114784 23078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77: Bootstrap starting.
I20260927 03:27:15.115603 23078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:15.115934 23078 log.cc:824] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:15.116901 23078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77: No bootstrap required, opened a new log
I20260927 03:27:15.118310 23078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } }
I20260927 03:27:15.118479 23078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533e2a6a7b994c0ea8b7d2f385201c77, State: Initialized, Role: FOLLOWER
I20260927 03:27:15.118564 23078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } }
I20260927 03:27:15.118945 23078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:27:15.118986 23052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 03:27:15.119045 23052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:15.119858 23052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } }
I20260927 03:27:15.120107 23052 leader_election.cc:288] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [CANDIDATE]: Term 1 election: Requested vote from peers 501881b8e7e04538a2527b5a4ce2e848 (127.0.98.61:45765), 45336c89a11a40608dbd4a31cb418367 (127.0.98.60:46691)
I20260927 03:27:15.120802 23127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "501881b8e7e04538a2527b5a4ce2e848"
I20260927 03:27:15.120951 23127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:15.122195 23127 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533e2a6a7b994c0ea8b7d2f385201c77 in term 1.
I20260927 03:27:15.123318 23202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "45336c89a11a40608dbd4a31cb418367"
I20260927 03:27:15.123442 23202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:15.124270 23202 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533e2a6a7b994c0ea8b7d2f385201c77 in term 1.
I20260927 03:27:15.118920 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } } }
I20260927 03:27:15.124584 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:27:15.123953 23274 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 03:27:15.124648 23026 leader_election.cc:302] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45336c89a11a40608dbd4a31cb418367, 533e2a6a7b994c0ea8b7d2f385201c77; no voters:
I20260927 03:27:15.124768 23263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:27:15.124914 23263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [term 1 LEADER]: Becoming Leader. State: Replica: 533e2a6a7b994c0ea8b7d2f385201c77, State: Running, Role: LEADER
I20260927 03:27:15.125010 23263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } }
W20260927 03:27:15.124679 23274 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:27:15.125487 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 533e2a6a7b994c0ea8b7d2f385201c77. Latest consensus state: current_term: 1 leader_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } } }
I20260927 03:27:15.125583 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:15.134522 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.125423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.0.98.60:46691)
0927 03:27:15.125466 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.125469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.126259 (+ 790us) server_negotiation.cc:407] Connection header received
0927 03:27:15.126358 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.126359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.126369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.126398 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.128080 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.128838 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.129900 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.130181 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.130219 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.131835 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:15.131836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:15.131837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:15.131843 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:15.133373 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.133731 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.133732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.133732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.133783 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.133960 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.133961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.133961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.134079 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:15.134159 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:15.134366 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 03:27:15.134438 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:27:15.132757 23282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:27:15.135655 23282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:27:15.143620 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.128467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.0.98.61:45765)
0927 03:27:15.128514 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.128516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.136616 (+ 8100us) server_negotiation.cc:407] Connection header received
0927 03:27:15.136708 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.136709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.136721 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.136747 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.137429 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.138142 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.139151 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.139436 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.139476 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.140942 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:15.140943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:15.140943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:15.140951 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:15.142409 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.142771 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.142772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.142772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.142854 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.143048 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.143048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.143049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.143174 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:15.143241 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:15.143452 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 03:27:15.143527 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:27:15.143908 23202 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367 [term 1 FOLLOWER]: Refusing update from remote peer 533e2a6a7b994c0ea8b7d2f385201c77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:27:15.144687 23263 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:27:15.145082 23127 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848 [term 1 FOLLOWER]: Refusing update from remote peer 533e2a6a7b994c0ea8b7d2f385201c77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:27:15.145364 23263 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:27:15.146533 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 533e2a6a7b994c0ea8b7d2f385201c77. Latest consensus state: current_term: 1 leader_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } } }
I20260927 03:27:15.146623 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:27:15.147805 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 533e2a6a7b994c0ea8b7d2f385201c77. Latest consensus state: current_term: 1 leader_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } } }
I20260927 03:27:15.147891 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:27:15.150660 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:15.128278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.0.98.62:41769)
0927 03:27:15.128319 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.128320 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.134962 (+ 6642us) server_negotiation.cc:407] Connection header received
0927 03:27:15.135018 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.135019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.135032 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.135084 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.135726 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.136432 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.137481 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:15.137788 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:15.137830 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:15.147756 (+ 9926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:15.147759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:15.147759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:15.147767 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:15.149368 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.149720 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.149721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.149722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.149771 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:15.149938 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:15.149939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:15.149939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:15.150067 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:15.150147 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:15.150506 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 03:27:15.150576 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 03:27:15.153602 23106 tablet.cc:2369] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:27:15.153635 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } } }
I20260927 03:27:15.153708 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:15.153735 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } } }
I20260927 03:27:15.153831 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:27:15.154088 23287 mvcc.cc:205] Tried to move back new op lower bound from 7333804585545375744 to 7333804585472053248. Current Snapshot: MvccSnapshot[applied={T|T < 7333804585545375744}]
I20260927 03:27:15.154206 23286 mvcc.cc:205] Tried to move back new op lower bound from 7333804585545375744 to 7333804585472053248. Current Snapshot: MvccSnapshot[applied={T|T < 7333804585545375744}]
I20260927 03:27:15.155000 23280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } } }
I20260927 03:27:15.155089 23280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77 [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:15.155512 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533e2a6a7b994c0ea8b7d2f385201c77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41769 } } peers { permanent_uuid: "501881b8e7e04538a2527b5a4ce2e848" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45765 } } peers { permanent_uuid: "45336c89a11a40608dbd4a31cb418367" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46691 } } }
I20260927 03:27:15.155593 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:27:15.161773 23282 catalog_manager.cc:1347] Generated new cluster ID: d20ddea0b7504c7f9f59fec1cb569705
I20260927 03:27:15.161829 23282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:27:15.185768 23282 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:27:15.186803 23282 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:27:15.190920 23282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 533e2a6a7b994c0ea8b7d2f385201c77: Generated new TSK 0
I20260927 03:27:15.191186 23282 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:27:15.217262 23081 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:15.216023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.0.98.62:41769)
0927 03:27:15.216069 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.216071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.216640 (+ 569us) server_negotiation.cc:407] Connection header received
0927 03:27:15.216736 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.216737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.216748 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.216775 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.217189 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46206: BlockingRecv error: recv got EOF from 127.0.0.1:46206 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 03:27:15.218405 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:15.216353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.0.98.61:45765)
0927 03:27:15.216405 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.216407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.217657 (+ 1250us) server_negotiation.cc:407] Connection header received
0927 03:27:15.217962 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.217963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.217973 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.218004 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.218306 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33170: BlockingRecv error: recv got EOF from 127.0.0.1:33170 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 03:27:15.219468 23233 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:15.217822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43378 (local address 127.0.98.60:46691)
0927 03:27:15.217858 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.217860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.218878 (+ 1018us) server_negotiation.cc:407] Connection header received
0927 03:27:15.219025 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.219027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.219037 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.219063 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.219407 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43378: BlockingRecv error: recv got EOF from 127.0.0.1:43378 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 03:27:15.221881 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:15.221090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33174 (local address 127.0.98.61:45765)
0927 03:27:15.221170 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.221172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.221395 (+ 223us) server_negotiation.cc:407] Connection header received
0927 03:27:15.221497 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.221498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.221516 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.221545 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.221830 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33174: BlockingRecv error: recv got EOF from 127.0.0.1:33174 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:27:15.224737 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:15.224068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.0.98.61:45765)
0927 03:27:15.224098 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.224100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.224305 (+ 205us) server_negotiation.cc:407] Connection header received
0927 03:27:15.224394 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.224395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.224404 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.224430 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.224690 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33180: BlockingRecv error: recv got EOF from 127.0.0.1:33180 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 03:27:15.228456 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:15.227787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.0.98.61:45765)
0927 03:27:15.227817 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.227819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.228034 (+ 215us) server_negotiation.cc:407] Connection header received
0927 03:27:15.228112 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.228113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.228122 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.228146 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.228405 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33196: BlockingRecv error: recv got EOF from 127.0.0.1:33196 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260927 03:27:15.233909 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:15.233439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.0.98.61:45765)
0927 03:27:15.233472 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.233474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.233545 (+ 71us) server_negotiation.cc:407] Connection header received
0927 03:27:15.233598 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.233599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.233608 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.233650 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.233868 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33208: BlockingRecv error: recv got EOF from 127.0.0.1:33208 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:27:15.243546 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:15.242891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.0.98.61:45765)
0927 03:27:15.242940 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.242943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.243068 (+ 125us) server_negotiation.cc:407] Connection header received
0927 03:27:15.243124 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.243125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.243137 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.243185 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.243490 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33224: BlockingRecv error: recv got EOF from 127.0.0.1:33224 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 03:27:15.261329 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:15.260688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.0.98.61:45765)
0927 03:27:15.260734 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.260736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.260836 (+ 100us) server_negotiation.cc:407] Connection header received
0927 03:27:15.260908 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.260908 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.260921 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.260961 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.261266 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33226: BlockingRecv error: recv got EOF from 127.0.0.1:33226 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:27:15.295017 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:15.294378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.0.98.61:45765)
0927 03:27:15.294447 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.294450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.294482 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:27:15.294550 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.294551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.294564 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.294645 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.294954 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33242: BlockingRecv error: recv got EOF from 127.0.0.1:33242 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 03:27:15.360680 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:15.360117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33256 (local address 127.0.98.61:45765)
0927 03:27:15.360176 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.360178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.360208 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:27:15.360271 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.360272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.360283 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.360346 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.360618 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33256: BlockingRecv error: recv got EOF from 127.0.0.1:33256 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 03:27:15.490512 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:15.489810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.0.98.61:45765)
0927 03:27:15.489874 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.489876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.489959 (+ 83us) server_negotiation.cc:407] Connection header received
0927 03:27:15.490038 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.490039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.490050 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.490096 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.490448 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33260: BlockingRecv error: recv got EOF from 127.0.0.1:33260 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 03:27:15.748412 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:27:15.747726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.0.98.61:45765)
0927 03:27:15.747789 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:15.747791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:15.747801 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:27:15.747896 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:15.747897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:15.747912 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:15.747961 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:15.748345 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33262: BlockingRecv error: recv got EOF from 127.0.0.1:33262 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:27:16.071132 23246 catalog_manager.cc:1259] Loaded cluster ID: d20ddea0b7504c7f9f59fec1cb569705
I20260927 03:27:16.071224 23246 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848: loading cluster ID for follower catalog manager: success
I20260927 03:27:16.072046 23246 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848: acquiring CA information for follower catalog manager: success
I20260927 03:27:16.072443 23246 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 501881b8e7e04538a2527b5a4ce2e848: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:27:16.089968 23260 catalog_manager.cc:1259] Loaded cluster ID: d20ddea0b7504c7f9f59fec1cb569705
I20260927 03:27:16.090040 23260 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367: loading cluster ID for follower catalog manager: success
I20260927 03:27:16.090871 23260 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367: acquiring CA information for follower catalog manager: success
I20260927 03:27:16.091135 23260 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 45336c89a11a40608dbd4a31cb418367: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:27:16.264142 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:16.261505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33276 (local address 127.0.98.61:45765)
0927 03:27:16.261617 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:16.261619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:16.261688 (+ 69us) server_negotiation.cc:407] Connection header received
0927 03:27:16.261765 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:16.261766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:16.261819 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:16.261915 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:16.262063 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:16.262560 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:16.263379 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:16.263582 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:16.263639 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:16.263805 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:16.263925 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:16.263994 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:16.264028 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 03:27:16.264051 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260927 03:27:16.268078 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:16.265811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43380 (local address 127.0.98.60:46691)
0927 03:27:16.265899 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:16.265901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:16.265909 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:27:16.265948 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:16.265949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:16.265981 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:16.266050 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:16.266122 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:16.266628 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:16.267377 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:16.267593 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:16.267639 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:16.267775 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:16.267888 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:16.267943 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:16.267972 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 03:27:16.267991 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":44,"threads_started":1}
I20260927 03:27:16.272286 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:16.269833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43392 (local address 127.0.98.60:46691)
0927 03:27:16.269880 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:16.269883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:16.269969 (+ 86us) server_negotiation.cc:407] Connection header received
0927 03:27:16.270069 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:16.270070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:16.270081 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:16.270132 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:16.270258 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:16.270768 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:16.271673 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:16.271866 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:16.271925 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:16.272057 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:16.272090 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:16.272137 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:16.272189 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 03:27:16.272209 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:27:16.273052 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:16.269757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46212 (local address 127.0.98.62:41769)
0927 03:27:16.269911 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:16.269913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:16.269922 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:27:16.270031 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:16.270032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:16.270086 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:16.270158 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:16.270336 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:16.271135 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:16.271999 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:16.272281 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:16.272336 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:16.272676 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:16.272790 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:16.272842 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:16.272946 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 03:27:16.272968 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":46,"threads_started":1}
I20260927 03:27:16.273530 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:16.269936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.0.98.61:45765)
0927 03:27:16.269983 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:16.269985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:16.270148 (+ 163us) server_negotiation.cc:407] Connection header received
0927 03:27:16.270223 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:16.270224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:16.270235 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:16.270284 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:27:16.270371 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:16.271136 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:16.272842 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:16.273104 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:16.273156 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:16.273312 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:27:16.273344 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:27:16.273424 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:16.273438 (+ 14us) server_negotiation.cc:299] Negotiation successful
0927 03:27:16.273459 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:27:16.274580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 23010
I20260927 03:27:16.279670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 23085
I20260927 03:27:16.284344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzllPpO/build/release/bin/kudu with pid 23160
2026-09-27T03:27:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2705 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5451 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (267064 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST